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SUMMARY 


The  present  study  set  out  to  perform  an  analysis  of 
the  field  evaluation  system  used  to  obtain  data  to  provide 
feedback  to  the  instructional  system  regarding  how  well 
graduates  of  resident  courses  were  performing  as  appren- 
tice level  specialists  on  their  first  duty  assignment 
following  completion  of  the  resident  course.  This  was 
accomplished  through  a series  of  consultations  and  visits 
with  personnel  at  the  technical  training  centers  and  at 
Headquarters  Air  Training  Command.  A list  of  field  gen- 
erated system  weaknesses  were  delineated  from  these  visits. 
Further  discussions  among  the  project  staff,  the  Air  Force 
Human  Resources  Laboratory  monitor,  and  personnel  from 
Headquarters,  Air  Training  Command  focussed  on  feasible 
alternatives  to  pursue. 

The  project  staff  developed  survey  instruments  which 
included  a basic  supervisor  questionnaire,  a basic  graduate 
questionnaire,  three  supplemental  supervisor  questionnaires, 
and  a set  of  diagnostic  task  follow-up  sheets.  These 
questionnaires  were  administered  through  the  Training  Eval- 
uation Division,  3350th  Technical  Training  Wing,  to  a sample 
of  186  graduates  of  five  classes  of  Course  3 ABR  42330  and 
to  their  supervisors.  With  one  exception,  the  questionnaires 
were  returned  from  the  field  to  the  Training  Evaluation 
Division  and  were  then  forwarded  to  the  project  staff  for 
processing.  After  the  project  staff  conducted  a series  of 
analyses  designed  to  provide  a more  in-depth  examination 
than  is  usually  accomplished,  the  completed  field  data 
forms  were  returned  by  the  project  staff  to  the  Training 
Evaluation  Division. 

The  relationship  of  ability  measures  to  various  cri- 
teria were  presented  both  in  terms  of  correlation  coeffi- 
cients and  in  tabular  format.  Results  indicate  that 
manipulation  of  ability  levels  would  not  improve  the 
situation  revealed  by  the  field  survey.  Reactions  of  both 
graduates  and  supervisors  indicate  that  the  survey  forms 
and  procedures  were  well  understood  and  provided  valid 
data . 


Job  cluster  analyses  revealed  some  cluster  differences 
although  they  tended  to  be  relatively  small.  Some  tasks, 
however,  were  associated  mainly  with  specific  clusters. 
Overall,  the  specialty  used  in  the  study  is  relatively 
homogeneous  as  Air  Force  specialties  go,  so  the  applica- 
tion of  these  analyses  to  other  specialties  may  reveal 
dramatically  different  results. 
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Reports  of  overall  difficulties  on  the  job  or  with 
OJT  by  graduates  and  supervisors  did  not  always  agree. 

Most  disagreements  were  in  the  direction  of  the  graduate 
perceiving  a difficulty  but  the  supervisor  not  reporting 
any  difficulty.  The  usual  practice  of  using  only  super- 
visor data  may  therefore  tend  to  understate  the  amount 
of  difficulty  the  graduates  actually  have. 

A large  number  of  tasks  did  not  meet  the  80%  satis- 
factory guideline  but  closer  examination  showed  that  the 
standard  acceptance  limits  used  by  the  project  staff  were 
higher  than  those  stated  by  the  Specialty  Training  Stan- 
dard for  some  of  the  tasks.  Furthermore,  for  all  tasks 
between  48%  and  84%  of  the  sample  had  not  performed  the 
task.  Comparisons  between  graduate  and  supervisor  reports 
of  tasks  performed  showed  more  agreement  than  disagree- 
ment but  there  were  considerable  discrepancies.  The  most 
frequently  cited  reason  for  non-performance  was  Not  Yet 
But  Will  Be.  These  tasks  should  therefore  undoubtedly 
be  retained  as  training  requirements.  However,  one  may 
tentatively  question  the  strength  of  the  training  require- 
ment for  those  tasks  for  which  there  were  large  numbers 
of  not  required  responses  due  to  situational  characteris- 
tics of  particular  duty  assignments. 

Supervisor  ratings  based  on  own  observations,  both 
task  specific  and  general  impressions,  were  somewhat  higher 
than  ratings  based  on  information  obtained  from  other 
personnel  at  the  work  site.  The  measures  of  overall  pro- 
ficiency provided  by  the  supervisor  data  were  highly  cor- 
related with  the  mean  of  their  individual  task  ratings, 
that  is,  the  general  overall  rating  made  by  the  super- 
visors was  much  the  same  as  a calculated  mean  rating  based 
on  the  individual  tasks  each  graduate  had  performed. 

The  supplemental  questionnaires  provided  different 
but  related  task  importance  indices.  The  relationship 
between  ;these  measures  and  general  criteria  of  job  per- 
formance indicated  statistically  significant  relationships 
in  the  directions  expected.  The  data  base  demonstrated 
an  internally  consistent  statistical  structure. 

Simple  decision  table  formatting  of  two  dimension 
comparisons  was  used  to  produce  gross  indices  of  adequacy 
and  to  identify  specific  tasks  in  terms  of  these  indices. 
These  data  were  summarized  as  an  input  to  personnel  at 
the  technical  training  center  for  detailed  analyses  of 
any  task  which  appeared  to  be  troublesome. 
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Overall,  the  procedures  and  instructions  utilized  in 
the  present  study  produced  results  which  members  of  the 
project  staff  feel  would  be  useful  to  curriculum  designers 
and  instructional  personnel  at  the  technical  training 
wings.  Before  any  recommendations  for  system-wide  changes 
are  made,  however,  these  results  should  be  submitted  to 
personnel  at  all  of  the  technical  training  centers  for 
review  and  comment  on  how  useful  they  could  be  for  the 
courses  involved  in  their  own  technical  training  center. 
This  review  process  should  involve  personnel  from  the 
centers’  command  staff,  the  Training  Evaluation  Division, 
and  curriculum  design  and  instructional  units.  Only  after 
a systematic  field  review  of  the  results  of  this  study 
would  recommendations  for  or  implementation  of  system- 
wide  changes  seem  warranted. 
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FIELD  EVALUATION  SYSTEM  FOR  AIR  FORCE  TECHNICAL  TRAINING: 
ANALYSIS  AND  MODIFICATION 


I.  INTRODUCTION 
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Technical  Training  in  the  Air  Force  is  a large  opera- 
tion. The  scope  of  the  technical  training  program  includes 
the  conduct  of  over  1000  different  resident  courses  which 
turn  out  over  180,000  graduates  annually.  The  overall 
technical  training  program  includes  virtually  all  Air 
Force  training  other  than  basic  military  training,  air- 
crew training,  and  professional  education.  Technical 
training,  which  is  primarily  vocational  education,  is 
achieved  not  only  through  the  resident  courses  mentioned 
above  but  also  in  large  part  through  an  organized,  dual- 
channel, on- the- j ob- training  program  in  which  the  majority 
of  enlisted  personnel  are  engaged  during  their  first  tour 
in  the  Air  Force.  This  study  was  concerned  primarily  with 
the' f esident  course  portion  of  the  overall  training  pro- 
gram with  particular  emphasis  on  the  3 skill  (or  appren- 
tice) level  courses. 

A systematic  evaluation  system  has  been  in  existence 
for  some  years.  This  system  was  designed  to  be  an  inte- 
gral part  of  the  Instructional  System  Development  (ISD) 
model  which  guides  the  development,  administration,  and 
evaluation  of  all  technical  training  courses.  The  field 
evaluation  portion  of  this  system,  along  with  internal 
evaluation  by  training  personnel  at  the  five  Air  Training 
Command  (ATC)  technical  training  centers,  constitutes  the 
evaluation  step  of  the  ISD  model.  The  evaluation  step  is 
designed  to  ensure  that  job  performance  requirements  are 
being  accomplished  by  technical  training  graduates  and  to 
determine  supervisors'  and  graduates'  opirions  of  the 
training  program.  The  field  evaluation  is  usually  accom- 
plished through  the  administration  of  questionnaires  to 
the  supervisors  of  recent  course  graduates  and  to  the 
graduates  themselves.  The  primary  purpose  of  these  ques- 
tionnaires is  to  obtain  data  on  the  judged,  demonstrated 
proficiency  level  of  the  graduates  on  a list  of  relevant 
job  tasks  selected  from  the  appropriate  Specialty  Train- 
ing Standard  (STS)  and  the  judged  adequacy  of  training 
received  on  these  tasks.  The  average  demonstrated  pro- 
ficiency levels  are  then  compared  with  those  specified  in 
the  STS  in  order  to  identify  potential  training  deficien- 
cies. 
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A need  was  perceived  to  improve  the  existing  field 
evaluation  system  in  order  to  make  it  more  effective  for 
the  purpose  of  providing  information  on  the  adequacy  of 
training  received  by  students  engaged  in  technical  train- 
ing courses.  It  was  anticipated  that  modified  question- 
naire procedures  and  more  comprehensive  analyses  than 
presently  carried  out,  making  fuller  use  of  computer 
technology,  would  help  to  improve  the  system.  Any  pro- 
posed modifications  were  to  take  full  cognizance  of  con- 
straints imposed  by  the  amount  and  type  of  manpower  needed, 
costs  of  developing  and  implementing  modifications,  and 
the  hardware  capabilities  of  present  base  level  computing 
systems.  Major  sequential  activities  of  the  study  in- 
cluded an  analysis  of  the  present  system  in  order  to 
identify  perceived  inadequacies,  development  of  proposed 
modifications,  and  the  demonstration  of  the  proposed  mod- 
ifications on  one  technical  training  course. 

The  primary  emphasis  of  the  study  was  methodological 
rather  than  substantive.  The  systems  analysis  was  con- 
ducted on  an  ATC-wide  basis  to  identify  problems  common 
to  all  technical  training  courses.  Instruments  and  pro- 
cedures developed  for  field  evaluation  modifications  were 
restricted  to  those  which  could  be  applied  to  most,  if 
not  all,  technical  training  courses.  While  the  analyses 
conducted  during  the  study  utilized  only  data  collected 
from  the  specific  course  selected  for  demonstration  pur- 
poses, they  were  designed  to  be  generic  in  nature  and  to 
demonstrate  the  types  of  analyses  and  interpretations 
that  could  be  made  from  such  data.  In  other  words,  we 
did  not  set  out  to  do  a substantive  evaluation  of  the 
demonstration  course.  Rather,  we  set  out  to  demonstrate 
the  feasibility  of  collecting  certain  types  of  informa- 
tion, making  certain  generic  types  of  analyses,  and  point- 
ing out  potential  ways  in  which  such  summarized  data 
could  be  used  in  making  judgments  about  training  adequacy 
and  possible  areas  of  training  deficiency.  This  report 
sets  forth  results  of  the  data  collected  on  the  demon- 
stration course  and  indicates  some  possible  interpreta- 
tions that  could  be  made.  This  report  does  not  set  forth 
definitive  conclusions  about  existing  training  deficiencies 
nor  methods  for  their  remediation.  The  results  of  the 
study  were  made  available  to  the  Training  Evaluation 
Division  of  the  3350th  Technical  Training  Wing,  which  has 
jurisdiction  over  the  demonstration  course,  for  their  use 
in  making  such  determination. 
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II.  PROJECT  ACTIVITIES 


The  present  field  evaluation  of  formal  technical 
school  graduates  is  governed  by  several  regulations  and 
manuals.  Air  Force  Regulation  50-38,  Field  Evaluation  of 
Formal  School  Graduates,  establishes  the  requirement  for 
a program  of  evaluation  for  formal  courses.  Air  Training 
Command  Regulation  52-1,  Training  Evaluation  and  Course 
Reviews,  establishes  policies,  responsibilities,  proce- 
dures, and  guidance  for  conducting  internal  course  eval- 
uations, reviews,  and  field  evaluations  to  provide  feed- 
back for  improvement  of  instructional  systems.  Air 
Training  Command  Manual  52-334,  Graduate  Evaluation  Ques- 
tionnaire ATC,  establishes  policies,  responsibilities, 
and  procedures  for  operating  the  data  automation  portion 
of  the  graduate  evaluation  questionnaire  system.  A review 
of  these  basic  documents  provided  the  project  staff  with 
an  overview  of  the  present  system  and  how  it  was  designed 
to  operate.  These  documents  also  highlighted  the  major 
constraints  of  the  present  system.  The  project  staff 
also  reviewed  recent  articles  from  the  professional  lit- 
erature having  to  do  with  student  evaluation  and  relevant 
measurement  techniques. 


Following  the  review  of  the  documentation  and  liter- 
ature, the  systems  analysis  progressed  with  a series  of 
meetings  between  the  project  staff  and  Air  Force  personnel 
having  responsibility  for  the  development,  administration, 
evaluation,  and  management  of  technical  training  activi- 
ties. An  initial  meeting  was  held  with  the  project  moni- 
tor and  the  staff  of  the  Technical  Training  Division  of 
the  Air  Force  Human  Resources  Laboratory  at  Lowry  Air 
Force  Base.  This  meeting  served  to  further  orient  the 
project  staff  and  to  clarify  overall  goals  and  expecta- 
tions for  the  study.  The  project  staff  also  held  dis-  '! 

cussions  with  the  Commander  and  personnel  from  the 
Evaluation  Division  of  the  Technical  Training  Wing  located 
at  Lowry  which  provided  greater  insights  regarding  how 
the  existing  system  operated  at  this  organization  level. 


A subsequent  visit  was  made  to  Headquarters,  Air 
Training  Command  at  Randolph  Air  Force  Base  for  discussions 
with  personnel  whose  primary  responsibilities  were  for 
overall  management  of  Air  Force  training  activities  in 
order  to  meet  operational  requirements.  Discussions 


were  held  with  personnel  from  organizational  units  con- 
cerned with  program  planning,  training  systems  development, 
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automation  activities.  These  discussions  were  helpful 
in  pointing  out  some  of  the  ooncerns  of  these  higher 
organizational  level  training  managers.  To  some  extent 
these  concerns  were  similar  to  those  of  personnel  at  the 
training  wing  level  and  to  some  extent  they  were  different, 
reflecting  the  differences  in  basic  responsibilities  at 
the  two  levels.  The  consensus  seemed  to  be  that,  while 
potential  modifications  to  the  field  evaluation  system 
should  in  no  way  degrade  its  usefulness  for  training  wing 
personnel,  it  might  be  useful  to  include  new  or  modified 
elements  which  more  fully  met  the  perceived  needs  of 
Headquarters  ATC  personnel. 

While  in  the  San  Antonio  area,  the  project  staff 
also  held  discussions  with  personnel  from  the  Training 
and  Evaluation  Division  of  the  technical  training  wing, 
from  the  Standards  Evaluation  Division  of  the  Officer 
Training  School,  and  from  the  Occupational  Measurement 
Center  (OMC),  all  located  at  Lackland  Air  Force  Base. 

The  latter  organization  was  included  since  they  routinely 
collect  and  analyze  data  regarding  tasks  performed  by 
the  various  specialties  throughout  the  Air  Force.  These 
data  provide  an  important  input  to  the  development  of 
both  quantitative  and  qualitative  training  requirements 
for  other  units  within  the  Air  Training  Command.  Per- 
sonnel utilization  patterns  which  indicated  job  clusters 
within  specialty  areas  could  have  an  important  influence 
on  training  requirements  and  procedures.  There  seemed 
to  be  some  feeling  that  fuller  utilization  of  such  clus- 
ters in  the  field  evaluation  system  might  provide  useful 
information  for  training  course  modification.  The  OMC 
was  also  working  with  various  measures  of  task  importance 
that  had  been  developed  by  AFHRL/OR.  Since  the  present 
field  evaluation  system  is  geared  mainly  to  the  job  task 
level,  indices  for  weighting  the  relative  importance  of 
the  various  job  tasks  could  be  important  in  the  evalua- 
tion process.  Task  Difficulty  is  one  such  measure. 

Results  with  this  scale  appear  promising  and  such  data 
are  now  apparently  being  collected  routinely  by  OMC  during 
Occupational  Surveys.  Consequences  of  Inadeqaute  Per- 
formance and  Task  Delay  Tolerance  are  two  additional 
scales  on  which  some  work  has  been  done;  however,  results 
are  not  conclusive.  If  data  collected  by  the  OMC  are  to 
be  directly  used  by  the  technical  training  wings,  there 
must  be  a way  for  translating  between  the  job  tasks  used 
in  Occupational  Surveys  and  those  job  tasks  specified  in 
the  STS,  which  is  the  primary  document  that  guides  the 
technical  training  system.  Some  of  the  work  has  already 
been  done  in  mapping  the  direct  relationship  between  job 
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tasks  from  the  two  sources.  The  preliminary  results  indi- 
cate that  the  task  can  be  done  with  an  acceptable  degree 
of  reliability  by  senior  enlisted  subject  matter  specialists. 

Discussions  with  personnel  from  the  Training  Evalua- 
tion Division  and  the  Standards  Evaluation  Division  focused 
on  the  operation  of  the  present  system  and  its  perceived 
inadequacies,  as  well  as  on  possible  remedial  actions. 

While  the  primary  emphasis  for  the  study  was  on  enlisted 
training,  the  officer  course  was  included  largely  because 
it  was  located  at  the  same  facility  and  to  obtain  some 
insight  into  generalizations  applicable  across  the  full 
training  spectrum. 

The  systems  analysis  continued  with  a series  of  meet- 
ings at  other  technical  training  wings  to  determine  per- 
ceived inadequacies  of  the  present  system  and  to  obtain 
preliminary  suggestions  for  possible  modifications.  The 
Nominal  Group  Technique  was  used  for  several  of  these 
problem  definition  meetings.  A distinguishing  character- 
istics of  this  technique  is  that,  while  it  is  conducted 
as  a small  group  activity,  each  participant  is  forced  to 
commit  himself  at  various  steps  along  the  way  before  a 
point  is  opened  up  for  group  discussion.  This  makes  it 
difficult  for  one  or  two  dominant  participants  to  control 
the  results  of  the  meeting  and  makes  it  possible  to  derive 
consensus  conclusions  based  on  each  individual  member's 
contr ibut ions . 

Participants  in  the  Nominal  Group  sessions  were  first 
asked  to  individually  write  down  their  answer  to  the  ques- 
tion, What  do  you  think  are  inadequacies  of  the  present 
field  evaluation  system  for  technical  training  courses? 

Each  participant  was  successively  asked  for  one  inade- 
quacy, which  was  put  on  a flip  chart  by  the  project  staff 
member  who  chaired  the  meeting.  This  round  robin  activity 
continued  until  all  individually  produced  inadequacies 
were  displayed  on  the  flip  charts.  This  was  followed  by 
a discussion  restricted  at  this  point  to  clarification  of 
the  points  listed,  which  sometimes  led  to  a combining  of 
separate  items.  This  produced  a list  of  potential  inade- 
quacies, all  of  which  were  relatively  uniformly  under- 
stood by  all  participants.  Participants  were  then  asked 
to  individually  rate  these  items  in  terms  of  importance. 

In  round  robin  fashion,  these  individual  ratings  were 
reported  to  the  group  and  were  summarized  on  a flip  chart. 

A group  discussion  was  then  held  regarding  the  summarized 
ratings  with  particular  attention  being  paid  to  those 
items  which  showed  a wide  discrepancy  in  ratings.  All 
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participants  were  encouraged  to  speak  for  or  against  the 
weights  assigned.  After  this  discussion,  a final  rating 
of  the  importance  of  the  inadequacies  listed  was  taken. 

This  final  rating  process  was  also  individually  done. 

When  summarized,  these  final  ratings  provided  a weighted 
summary  of  the  conclusions  reached  by  the  group. 

Four  Nominal  Group  meetings  were  held  at  Lowry  and 
Chanute  Air  Force  Bases.  Each  group  contained  represen- 
tatives of  the  Evaluation  Division  and  curriculum  design 
and/or  instructional  units.  Additional  problem  definition 
sessions  were  held  with  similar  types  of  personnel  from 
the  technical  training  wings  at  Keesler  and  Sheppard  Air 
Force  Bases,  as  well  as  from  the  School  of  Health  Care 
Sciences  at  Sheppard.  While  there  were  differences  of 
opinion  expressed  at  the  various  training  sites,  there 
was  also  a considerable  amount  of  agreement  across  the 
various  technical  training  wings. 

Results  from  the  series  of  field  visits  conducted  by 
the  project  staff  identified  the  following  general  system 
elements  which  were  perceived  as  presenting  some  problems 
which  might  be  addressed  in  modifying  the  present  system. 

• Time  considerations 

Time  lag  between  when  a course  is  given  and  when 
the  final  Training  Evaluation  Report  reaches 
the  school  for  official  action 

Problems  inherent  with  the  assessment  of  graduates 
after  only  a short  period  of  time  on  the  job 

• Quality  of  data 

Questionnaire  responses  mediated  by  general 
perceptions  of  the  graduate,  not  necessarily  on 
the  basis  of  actual  observation  « 

Limited  use  of  write-in  capabilities  to  supple- 
ment rating  data 

Need  for  training  in  and  understanding  of  the 
purposes  of  the  field  evaluation  questionnaire 
by  the  supervisors  who  complete  them 
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• Field  Evaluation  Questionnaire 
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Questionnaire  content,  format,  and  item  wording 

Limiting  effects  of  present  answer  sheet  format 

Adequacy  of  instructions  to  questionnaire  respon- 
dents 

• Sampling  procedures 

Sampling  of  supervisors  only,  as  opposed  to 
graduates  also 

Representativeness  of  graduate  sample  in  regard 
to  population  variables  such  as  sex,  aptitude 
level,  and  educational  level 

• Feedback  procedures 

Limited  feedback  to  field  units  and  to  the  per- 
sons completing  the  questionnaires 

Limited  feedback  to  instructor  personnel 

• Analyses 

Descriptive  only,  with  limited  analytical  capa- 
bilities 

Data  not  relatable  to  specific  instruction 
activities  or  hardware  system  usage 

Limited  capabilities  for  job  cluster  comparisons 
within  specialty  analyses 

Limited  capabilities  for  displaying  longitudinal 
data 

• System  element  interfaces 

Role  of  instructional  personnel  vis-a-vis  evalua- 
tion personnel  in  the  evaluation  process 

Integration  of  Occupational  Survey  data,  STS 
data,  and  Training  Quality  Report  data 


14 


( 


• Role  of  the  training  evaluation  system 

Concept  of  quality  control  and  the  training 
contract 

Validity  of  the  requirements  set  by  the  STS 

Evaluation  of  resident  courses  in  an  isolated 
sense  or  as  part  of  a total  training  system  that 
includes  resident  course,  field  training,  and 
OJT  activities 

• Qualification  of  evaluation  personnel 

Limited  use  of  highly  experienced  subject  matter 
specialists  in  the  total  evaluation  process 

Type  of  personnel  making  field  observation  visits 

Using  this  list  of  field  generated  system  elements 
which  were  perceived  as  presenting  some  problems  as  a 
guideline,  the  project  staff  generated  a list  of  proposed 
activities  that  might  be  undertaken  within  the  scope  of 
the  contractual  effort  to  modify  procedures  and  instru- 
ments bearing  on  these  system  element's.  In  generating 
this  list,  the  project  staff  had  to  compromise  on  many 
occasions  between  judgments  of  what  might  be  theoretically 
desirable  and  what  might  be  feasible  within  the  scope  of 
the  present  study.  The  following  activities  were  included 
in  the  list  of  potential  activities. 

• Integration  of  system  data  elements 

Make  a comparison  of  STS  tasks  on  a task-fov- 
task  basis  with  the  Occupational  Survey  tasks 
and  the  statements  from  AFM  39-1 

• Instructional  analyses 

Allocate  Plan  of  Instruction  instructional 
blocks  to  tasks 

Carry  out  simple  decision  table  analyses  of 
instructional  emphasis  versus  various  measures 
of  task  importance  and  task  performance 

Determine  whether  a demonstration  of  criterion 
performance  for  each  task  was  required  during 
the  training  course 
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• Field  data  collection  procedures 

Use  of  two-step  procedure,  the  first  to  identify 
problem  tasks  and  the  second  to  diagnose  causes 
of  the  problem 

Use  of  "time  required"  and  "supervision  required" 
scales  as  well  as  the  present  scale  to  measure 
task  proficiency 

Use  of  confidence  level  indicators  along  with 
the  level  of  performance  scale 

Use  of  task  specific  diagnostic  questions 

Collection  of  task  criticality  data  from  super- 
visors 

Collection  of  supervisors'  opinions  regarding 
the  validity  of  present  STS  standards 

Collection  of  data  regarding  reasons  for  non- 
performance of  job  tasks  included  on  the  STS 

Collection  of  data  from  graduates  regarding  per- 
ceived adequacy  of  resident  training  for  job 
task  performance  and  OJT  activities 

• Analysis  activities 

Determine  overall  indices  of  training  adequacy 
using  simple  decision  table  formats 

Describe  sample  in  terms  of  wi thin- spec i al ty 
job  clusters 

Compare  task  performance  with  a variety  of 
relevant  variables  such  as  instructional  time, 
job  clusters,  ability  levels,  and  sex  if  rele- 
vant 

When  there  are  a substantial  number  of  tasks  that 
are  common  over  several  evaluation  periods,  make 
longitudinal  comparisons  to  ascertain  trends 

Display  various  indices  of  training  adequacy 
by  job-task  level  to  provide  indications  of  which 
tasks  warrant  in-depth  qualitative  analysis 
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This  list  of  proposed  potential  activities  provided 
the  agenda  for  a meeting  of  the  project  staff,  the  AFHRL 
contract  monitor,  and  personnel  from  Headquarters,  Air 
Training  Command  representing  program  planning;  technical 
training  management,  standards  and  evaluation;  and  data 
automation  support  Results  of  this  meeting  guided  the 
project  staff  in  developing  a series  of  instruments  and 
procedures  for  trial  use  on  a single  course.  The  course 
selected  by  Headquarters,  Air  Training  Command,  with  the 
concurrence  of  the  cognizant  technical  training  wing  and 
the  project  staff,  was  Course  3 ABR  42330,  Aircraft  Elec- 
trical Systems  Specialist,  which  was  conducted  by  the 
3350th  Technical  Training  Wing  at  Chanute  Air  Force  Base. 


III.  Instruments  and  Procedures  Used 

Immediately  after  the  selection  of  the  demonstration 
course,  the  project  staff  conferred  with  the  Training 
Evaluation  Division  of  the  3350th  Technical  Training  Wing 
in  order  to  coordinate  the  details  of  conducting  the 
study  within  the  evaluation  time  frame  previously  estab- 
lished for  the  course.  It  was  mutually  agreed  that  the 
survey  process  for  this  study  would  substitute  for  the 
normal  process  rather  than  be  added  to  it,  but  that  the 
modified  process  would  produce  all  of  the  types  of  data 
normally  collected,  as  well  as  those  especially  developed 
for  the  study.  The  basic  content  of  the  instruments  was 
supplied  by  the  Training  Evaluation  Division,  while  the 
format  of  the  instruments  and  subsequent  analyses  were 
the  responsibility  of  the  project  staff. 

Copies  of  the  five  survey  instruments  developed  for 
use  in  the  study  are  presented  in  Appendix  A.  A basic 
questionnaire  was  designed  for  completion  by  supervisors 
of  graduates.  This  was  to  be  completed  by  all  supervisors 
of  graduates  in  the  evaluation  sample.  Another  basic 
questionnaire  was  designed  for  completion  by  the  graduates 
themselves.  This  was  to  be  completed  by  all  graduates  in 
the  evaluation  sample.  Three  additional  supplemental 
questionnaires  were  designed  for  completion  by  supervisors 
of  graduates.  Since  the  purpose  of  their  use  in  the  study 
was  primarily  to  demonstrate  feasibility  of  their  use 
rather  than  to  provide  definitive  substantive  data,  each 
of  these  supplemental  questionnaires  was  sent  to  approxi- 
mately one-third  of  the  sample  of  supervisors.  The  eval- 
uation time  frame,  number  of  classes  included,  and  sample 
selection  within  each  class  was  all  accomplished  by  the 
Training  Evaluation  Division  in  accordance  with  their 
normal  evaluation  procedures. 
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The  project  staff  prepared  initial  drafts  of  proposed 
questionnaire  formats  using  somewhat  different  scaling  pro- 
cedures. For  the  basic  supervisor  questionnaire,  three 
different  scales  for  rating  proficiency  were  developed.  One 
was  the  present  scale  which  measures  proficiency  in  terms 
of  the  amount  of  the  total  task  that  can  be  performed.  A 
second  scale  measured  proficiency  in  terms  of  a time  dimen- 
sion to  perform  the  task  satisfactorily.  The  third  scale 
measured  proficiency  in  terms  of  the  amount  of  supervision 
needed  to  perform  the  task  satisfactorily.  For  the  basic 
graduate  questionnaire,  two  different  physical  rating  for- 
mats were  developed.  For  each  of  two  of  the  supplemental 
supervisor  questionnaires,  two  different  scales  were  devel- 
oped. For  the  third  supplemental  questionnaire,  a basic 
set  of  rating  categories  was  developed  as  a means  for 
generating  other  appropriate  categories.  Arrangements 
were  made  through  the  contract  monitor  to  submit  these 
draft  formats  to  a group  of  experienced  work  center  super- 
visors for  review  and  criticism.  Six  experienced  non- 
commissioned officers  from  the  89th  Organizational  Main- 
tenance Squadron,  and  five  experienced  non-commissioned 
officers  and  one  civilian  from  the  89th  Field  Maintenance 
Squadron,  from  the  89th  Military  Airlift  Group  at  Andrews 
Air  Force  Base  participated  in  a 2-hour  critique  session. 

The  project  staff  used  the  results  of  this  critique  to 
select  a single  scale  and  set  of  categories  for  use  on 
each  of  the  five  survey  instruments. 

The  final  draft  revisions  were  sent  to  the  Training 
Evaluation  Division  for  review  and  comment.  Final  survey 
forms  were  then  printed  by  the  Air  Force.  The  survey 
forms  were  mailed  to  graduates  and  supervisors  of  graduates 
in  the  evaluation  sample  by  the  Training  and  Evaluation 
Division  through  normal  Air  Force  channels.  The  survey 
forms  were  also  sent  back  to  the  Training  Evaluation  Division 
as  they  would  normally  be.  After  preliminary  perusal  by 
the  Training  Evaluation  Division,  the  completed  questionnaires 
were  sent  to  the  project  staff  for  processing. 

Distinguishing  features  of  the  survey  questionnaires 
used  in  the  study  include  the  following. 


Basic  Supervisor  Questionnaire 

The  basic  content  of  this  form  was  supplied  by  the 
Training  Evaluation  Division.  It  included  a set  of  back- 
ground questions  required  by  ATC  Manual  52-334,  as  well  as 
a few  specific  questions  regarding  the  utilization  of 
female  personnel  in  this  Air  Force  Specialty  Code.  It 
also  included  a question  developed  by  the  project  staff 


regarding  job  clusters  within  the  specialty.  The  question 
was  phrased  in  terms  of  identifying  groups  of  functional 
activities  in  which  the  most  time  was  spent.  Four  such 
job  clusters  were  represented  by  the  question  options. 

These  options  were  developed  by  the  project  staff  from 
data  supplied  by  experienced  work  center  supervisors. 

. Responses  to  this  question  allowed  for  analyses  in  terms 

of  job  clusters  within  specialties.  While  in  this  study 
the  clusters  were  developed  rationally  from  supervisors' 
opinions,  the  options  could  be  developed  directly  from 
empirically  derived  job  clusters  from  Occupational  Survey 
Reports.  These  data  are  normally  available  to  the  staff 
at  the  time  of  questionnaire  development  and  could  be  used 
in  development  of  the  appropriate  job  cluster  question 
options.  The  concept  of  job  cluster  analyses  is  discussed 
later  in  the  section  on  results. 

The  81  job  tasks  included  in  the  proficiency  rating 
section  were  supplied  by  the  Training  Evaluation  Division. 
They  basically  represented  those  tasks  from  the  Specialty 
Training  Standard  which  were  designated  as  a 1 or  2 pro- 
ficiency level.  This  selection  process  was  the  one  norm- 
ally used  to  select  job  tasks  for  inclusion  on  the  field 
evaluation  questionnaire. 

A primary  difference  between  normal  procedures  and  the 
procedures  used  in  this  study  was  that  in  this  study  the 
supervisors  made  all  of  their,  ratings  concerning  specific 
job  tasks  on  the  survey  form  itself  rather  than  recording 
such  ratings  on  a separate  answer  sheet  (Form  1612C). 

This  was  done  more  from  the  point  of  view  of  influencing 
the  rating  process  than  for  administrative  reasons.  The 
form  was  designed  so  that  the  supervisor  made  a series  of 
judgments  sequentially  about  each  task.  The  first  judg- 
ment was  a direct  question  as  to  whether  the  graduate  had 
performed  the  task  since  assignment  to  the  unit.  This  was 
put  first  in  the  series  in  an  attempt  to  get  a statement 
of  performance  or  non-performance  relatively  uncontaminated 
by  any  evaluative  connotations.  The  next  rating  required 
was  a conditional  one  depending  on  whether  the  task  was 
performed.  For  those  tasks  not  performed,  the  supervisor 
was  to  specify  the  major  reason  for  non-performance.  Five 
basic  reasons  had  been  developed  based  on  opinions  of 
experienced  work  center  supervisors,  and  the  rating  super- 
> sivor  was  merely  to  check  one  or  more  of  the  five.  These 

first  two  sections  of  the  task  ratings  were  designed  to 
create  a rating  framework  which  made  the  indication  of 
i non-performance  a legitimate  response.  One  rationale 

for  this  approach  was  that,  to  the  extent  possible,  the 
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proficiency  level  ratings  were  to  be  based  on  direct 
observations  of  performance.  The  other  rationale  was  that 
the  present  system  typically  indicates  a high  degree  of 
performance  on  all  tasks,  a result  which  seems  to  be 
logically  contraindicated  by  the  fact  that  the  ratings  are 
made  after  only  a short  time  on  the  job. 
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The  next  judgment  requested  was  for  an  indication  of 
the  major  basis  used  by  the  supervisor  in  making  the  pro- 
ficiency level  judgment.  The  categories  used  were  the 
fol lowing : 

Own  Task  observation.  Use  this  if  you  "ave  personally 
seen  the  graduate  perform  the  task  or  a highly  simi- 
lar task,  or  if  you  have  personally  seen  the  product 
of  his  work  on  this  task  or  a highly  similar  task. 

Task  information  from  others.  Use  this  if  you  have 
not  personal ly  seen  the  graduate  perform,  but  are 
basing  your  rating  on  information  you  got  from  other 
supervisors  or  experienced  journeymen  who  have  seen 
the  graduate  perform  this  task  or  a highly  similar 
task . 

General  impressions.  Use  this  if  you  have  not  ob- 
served the  graduate  actually  perform  this  or  a highly 
similar  task  and  are  basing  your  judgment  on  a general 
opinion  based  on  observation  of  overall  activities 
at  this  work  site. 

This  step  was  included  to  reinforce  the  notion  of  rating 
based  on  actual  observed  performance  and  to  provide  data 
to  determine  what  influence  the  basis  for  rating  had  on 
the  proficiency  levels  indicated. 

The  last  step  requested  a rating  of  the  proficiency 
level  demonstrated  by  the  graduate  on  the  job  task.  The 
scale  used  for  this  was  identical  to  that  normally  used, 
e.g..  Incapable,  Extremely  Limited,  Partially  Proficient, 
Competent,  and  Highly  Proficient. 

Basic  Graduate  Questionnaire 

Past  practice  in  evaluation  of  this  apprentice  level 
course  did  not  include  collecting  field  data  from  graduates 
themselves.  It  was  included  in  this  study  in  order  to 
obtain  direct  reports  of  the  opinions  of  graduates  as  to 
the  adequacy  of  the  resident  course  training  in  terms  of 
performance  on  the  tasks  they  had  performed,  in  terms  of 
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completion  of  the  relevant  Career  Development  Course  (CDC) 
in  which  most  were  enrolled,  and  in  terms  of  any  additional 
formal  Field  Training  Detachment  (FTD)  courses  in  which  they 
may  have  participated.  It  also  secondarily  provided  an 
index  of  task  performance  which  could  later  be  compared 
with  supervisor  reports  of  task  performance. 

The  graduate  questionnaire  contained  a few  background 
questions  called  for  by  ATC  Manual  52-334.  It  also  included 
a job  cluster  question  whose  options  were  identical  with 
those  used  in  the  supervisor  questionnaire.  In  the  case 
of  the  graduate,  multiple  responses  were  discouraged  but 
allowed  if  the  graduate  felt  he  had  spent  important  but 
equal  amounts  of  time  in  more  than  one  functional  area. 

The  graduate  questionnaire  contained  the  same  81  job 
tasks  as  on  the  supervisor  questionnaire.  For  each  job 
task,  the  graduate  was  asked  to  indicate  his  opinion 
separately  about  the  adequacy  of  the  resident  course 
training  received  for  job  task  performance,  for  CDC  work, 
and  for  FTD  work  if  applicable.  The  scale  used  for  all 
three  ratings  was  as  follows. 

1 = The  resident  course  did  not  provide  me  with 

the  level  of  knowledge  and  skill  I needed  to 
perform  assigned  job  tasks  at  the  level  expec- 
ted of  me,  or  to  complete  the  CDC  in  the  expec- 
ted time  period,  or  to  satisfactorily  complete 
FTD  courses  in  which  I participated. 

2 = The  resident  oourse  provided  me  with  the  level 

of  knowledge  and  skill  I needed  to  perform 
assigned  job  tasks  at  the  level  expected  of  me, 
or  to  complete  the  CDC  in  the  expected  time 
period,  or  to  satisfactorily  complete  FTD  courses 
in  which  I participated. 

3 = The  resident  course  provided  me  with  a higher 

level  of  knowledge  and  skill  than  I needed  to 
perform  assigned  job  tasks  at  the  level  expected 
of  me,  or  to  complete  the  CDC  in  the  expected 
time  period,  or  to  satisfactorily  complete  FTD 
courses  in  which  I participated. 

These  scale  values  translate  into  crude  indices  of  under- 
training,  appropriate  training,  and  overtraining  based  on 
the  opinions  of  the  relatively  inexperienced  course  gradu- 
ates. While  the  experience  level  of  the  graduates  is  a 
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weakness  of  this  type  of  data,  graduates  are  the  only 
possible  source  for  this  type  of  information  and  are  a 
valid  source  of  data  for  such  information  at  this  point 
in  time.  The  real  source  of  weakness  in  such  data  is 
inherent  in  the  practice  of  making  training  value  assess- 
ments after  such  a short  period  of  time  on  the  job,  and 
this  statement  applies  equally  to  the  data  derived  from 
supervisors.  Such  weakness  is  the  price  one  pays  for 
making  assessments  at  that  time  in  order  to  gain  the 
advantage  of  being  able  to  attribute  obtained  results  to 
the  resident  training  course  rather  than  to  intervening  job 
and  OJT  activities.  A logical  way  to  correct  for  this 
weakness  would  be  to  make  repeat  assessments  later  and 
to  compare  the  results  from  the  two  time  periods.  Obtained 
differences  using  such  an  approach  would  be  attributable 
to  a combination  of  factors  from  initial  training  and 
intervening  activities.  If  either  judgmental  or  empirical 
data  were  collected  regarding  these  factors,  estimates 
could  be  obtained  of  the  attributional  influence  of  them. 

It  was  not  possible  to  implement  this  approach  within 
the  constraints  of  the  contractual  effort. 

Supplemental  Supervisor  Questionnaires 

All  three  o£  the  supplemental  questionnaires  were 
designed  to  obtain  data  on  relevant  STS  tasks  not  asso- 
ciated directly  with  the  task  performance  of  graduates. 

Two  of  the  questionnaires  were  designed  to  generate 
indices  of  task  importance.  The  third  was  designed  to 
confirm  the  validity  of  the  stated  STS  requirements.  These 
data  were  deemed  to  be  useful  for  two  majoT  reasons. 

First,  whenever  one  works  with  a large  number  of  job 
tasks  within  a specialty,  importance  weights  help  one  to 
focus  on  the  most  important  tasks.  If  there  are  constraints 
which  limit  the  number  of  tasks  which  can  be  given  full 
attention,  the  importance  weights  indicate  which  tasks 
should  receive  priority. 

The  second  reason  for  the  usefulness  of  these  data 
has  to  do  with  the  time  factor  in  making  evaluations  and 
providing  the  results  to  the  instructional  system.  If  the 
evaluation  process  is  restricted  solely  to  indices  derived 
from  graduates'  after-training  job  performance,  there  is 
simply  no  way  that  the  present  time  period  can  be  shortened 
appreciably  and  still  have  results  based  on  acceptable 
sample  sizes.  The  amount  of  time  it  takes  to  aggregate 
adequate  samples  depends  on  student  flow,  but  even  in 
high  flow  courses  it  takes  many  months  to  build  up  respec- 
table samples.  On  lower  flow  courses,  the  time  has  to 
be  longer.  If  the  automated  data  support  system  for  the 
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field  evaluation  program  were  such  that  analyses  could  be 
done  on  an  incremental  basis,  class  by  class,  some  evalu- 
ative information  could  be  generated  for  a course  in  a 
shorter  time  frame.  Such  data  would,  of  course,  be  more 
tentative  in  nature  because  they  would  be  initially  based 
on  input  from  only  one  class  and  would  slowly  build  up  on 
a class  by  class  basis  until  a respectable  sample  size 
was  obtained.  Such  a procedure  is  currently  within  the 
hardware  capabilities  of  the  present  data  support  system 
but  the  administrative  feasibility  of  such  an  approach  is 
doubtful,  particularly  when  the  cost/benefit  ratio  is 
considered . 

Measures  other  than  graduate  performance,  however, 
can  be  used  to  evaluate  certain  portions  of  the  training 
process.  Once  the  basic  training  requirements  have  been 
specified  and  prioritized,  one  can  examine  the  instructional 
program  itself  to  determine  the  relative  amounts  of  instruc- 
tional time  devoted  to  each  task  and  evaluate  this  allo- 
cation in  terms  of  the  prioritized  requirements.  In  the 
Air  Force  system,  the  STS  sets  forth  the  basic  require- 
ments, the  Program  of  Instruction  sets  forth  instructional 
time  allocations  to  the  various  tasks;  and  in  the  present 
study,  two  of  the  supplemental  questionnaires  supply 
indices  for  prioritizing  these  requirements  in  terms  of 
importance.  Time  allocations  can  thus  be  compared  to 
task  importance  dimensions  to  generate  gross  indices  of 
apparent  over-and  under- emphas is . This  was  done  in  the 
present  study  and  will  be  discussed  later  in  this  report. 
While  the  supplemental  questionnaires  were  associated 
with  the  field  evaluation  process  in  the  present  study, 
they  need  not  be.  Since  they  are  not  associated  directly 
with  graduate  performance,  they  could  be  administered 
whenever  and  as  frequently  as  desired  to  provide  up-to- 
date  indices  of  task  important.  Such  procedures  and 
analyses  can  circumvent  the  time  lag  problem  and  supply 
at  least  partial  evaluation  of  the  training  course  on  a 
more  timely  basis. 

Supplemental  Questionnaire  A - Task  Criticality: 

This  questionnaire  presented  the  supervisor  with  a list 
of  relevant  tasks.  For  each  task,  the  supervisor  was 
asked  to  make  a judgment  of  the  task's  criticality  for 
apprentice  level  specialists  using  the  following  scale: 

0 = Task  is  not  particularly  relevant  to  the  air- 
man's assignment  in  that  it  is  rarely  if  ever 
performed  by  an  incumbent  at  the  apprentice 
level  . 
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1 = Task  is  of  limited  relevance  to  the  airman's 

assignment.  It  is  nice-to-know  and  useful  but 
not  essential  in  that  an  airman  can  satisfac- 
torily perform  his  duty  assignment  without  being 
able  to  perform  it  satisfactorily. 

2 = Task  is  important  to  th«  airman's  assignment 

in  that  ability  to  perform  it  satisfactorily 
will  enhance  mission  accomplishment  of  the  unit 
to  which  he  is  assigned. 

3 = Task  is  critical  to  the  airman's  assignment  in 

that  mission  accomplishment  of  the  unit  to  which 
he  is  assigned  cannot  be  attained  if  he  cannot 
perform  it  satisfactorily. 

Supplemental  Questionnaire  B - Training  Level  Standards: 
This  questionnaire  presented  the  supervisors  with  the  same 
set  of  relevant  STS  tasks  with  an  indication  of  the  level 
of  proficiency  called  for  at  the  apprentice  level.  For 
each  task,  the  supervisor  was  asked  to  indicate  his  judg- 
ment of  whether  the  proficiency  level  specified  was  a 
realistic  expectation  for  apprentices  approximately  90 
to  120  days  after  completion  of  a basic  resident  training 
course.  If  the  supervisor  did  not  agree  with  the  stated 
level,  he  was  asked  to  provide  the  appropriate  level. 
Supervisors  were  asked  to  "keep  in  mind  current  manpower 
and  budgetary  constraints  when  making  your  judgments." 

This  type  of  data  was  designed  to  confirm  the  train- 
ing level  standards  currently  in  being.  Since  the  initial 
statement  of  training  requirements  is  such  an  important 
part  of  any  overall  instructional  system  and  since  some 
questions  had  been  raised  during  the  early  field  visits 
about  the  manner  in  which  the  STS  were  coordinated  through- 
out the  Air  Force,  it  was  deemed  useful  to  include  this 
confirmation  process  in  the  present  study.  It  is  partic- 
ularly appropriate  to  the  advocates  of  the  quality  control 
concept  as  almost  the  sole  purpose  for  field  evaluation. 

These  advocates  argue  that  the  STS  sets  forth  the  "training 
contract"  between  the  Technical  Training  Wings  and  their 
Air  Force  users  and  that  the  only  legitimate  purpose  of 
field  evaluation  is  to  check  on  performance  against  that 
contract.  The  type  of  data  collected  merely  attempts  to 
reconfirm  the  validity  of  the  "contract  terms"  that  were 
previously  established. 
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Supplemental  Questionnaire  C - Impact  of  Training 
Deficiencies : This  questionnaire  also  contained  the  same 

list  of  relevant  tasks  taken  from  the  STS  as  those  on 
the  other  supplemental  questionnaires  along  with  the 
present  level  of  proficiency  required  for  the  apprentice 
level.  For  each  task,  the  supervisor  was  asked  to  indi- 
cate his  judgment  as  to  what  impact  there  would  be  on  the 
unit  if  apprentice  level  specialists  did  not  in  fact  demon- 
strate that  level  of  proficiency.  Eight  categories  were 
included  on  the  questionnaire  and  the  supervisor  was  asked 
to  check  any  and  all  that  applied  to  a given  task.  The 
categories  which  were  derived  from  data  obtained  from  a 
group  of  experienced  work  center  supervisors  were  the 
following: 

Not  required.  No  impact  since  the  task  is  not 
required  by  the  incumbent  of  this  particular  duty 
assignment. 

No  impact.  The  task  is  required  but  performance 
deficiencies  of  apprentices  would  have  no  appre- 
ciable impact  on  the  unit. 

Upgrading  impossible.  Without  the  level  specified, 
the  incumbent  could  not  handle  required  CDC  and  OJT 
assignments  in  order  to  upgrade  to  the  5-skill  level. 

Upgrading  delayed.  The  incumbent  could  upgrade  to 
the  5-skill  level  blit  would  take  longer  than  usual. 

Time  of  others.  Time  of  present  5-,  7-,  and  9-  level 
personnel  normally  available  for  operational  duties 
would  be  diverted  in  order  to  provide  additional 
training  or  supervision  of  the  apprentices. 

Reduced  reliability.  Reliability  of  unit  hardware 
and/or  weapon  systems  would  be  reduced. 

Increased  downtime.  Out- o f - s erv ic e time  of  unit 
hardware  and/or  weapon  systems  would  be  increased. 

Delayed  schedules.  Unit  production  schedules  would 
likely  be  delayed. 

The  data  provided  from  this  questionnaire  were  used 
to  generate  one  index  of  task  importance.  This  index 
was  also  specifically  included  in  the  study  in  an  attempt 
to  gather  data  on  the  question  raised  by  personnel  at 
Headquarters,  Air  Training  Command  regarding  what  it 
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costs  the  Air  Force  if  stated  training  requirements  are 
not  achieved  by  resident  level  courses. 

A short  Reactions  to  the  Survey  section  was  included 
at  the  end  of  each  of  the  five  questionnaires  used  in 
the  study.  Adequate  space  was  also  provided  for  open- 
ended  write-in  comments  on  each  of  the  questionnaires. 

Diagnostic  Questionnaire 

The  basic  supervisor  and  graduate  questionnaires 
provided  data  that  were  primarily  helpful  in  identifying 
tasks  on  which  individual  graduates  had  difficulty  but 
provided  little  information  regarding  apparent  causes  for 
that  difficulty.  In  an  attempt  to  gather  more  diagnostic 
information,  a set  of  81  follow-up  evaluation  sheets  was 
prepared,  one  for  each  of  the  81  job  tasks  included  on  the 
supervisor  and  graduate  questionnaires.  While  the  ques- 
tions for  these  follow-up  questionnaires  were  slightly 
different  from  task  to  task,  reflecting  the  nature  of 
the  task  itself,  each  task  sheet  contained  questions 
designed  to  determine  the  following: 

Precisely  what  tools,  equipment,  and  procedures 
are  involved  in  the  performance  deficiency 

Was  lack  of  task  related  knowledge  judged  to  be 
the  primary  cause  of  performance  deficiency 

Was  lack  of  ability  to  apply  acquired  knowledge 
in  a practical  situation  (skill)  judged  to  be  the 
primary  cause  of  performance  deficiency 

Were  motivational  or  attitudinal  factors  judged 
to  be  the  primary  cause  of  performance  deficiency 

Adequate  space  for  open-ended  comments  was  also  provided 
on  each  task  follow-up  sheet. 

As  completed  supervisor  questionnaires  were  received, 
the  project  staff  reviewed  each  to  determine  if  it  con- 
tained any  proficiency  ratings  of  either  Incapable  or 
Extremely  Limited.  A set  of  follow-up  diagnostic  sheets 
was  sent  to  all  supervisors  who  had  rated  one  or  more  tasks 
as  less  than  Partially  Proficient.  The  number  of  sheets 
in  any  one  set  was  equal  to  the  number  of  tasks  so  rated. 

A mailing  address  of  the  supervisor  completing  the  basic 
questionnaire  had  been  requested  and  these  were  used  to 
mail  the  sets  of  diagnostic  questionnaires.  It  was  not 
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expected  that  th.re  would  be  sufficient  numbers  of  returns 
on  each  of  the  separate  81  job  tasks  for  elaborate  statis- 
tical summaries,  but  it  was  anticipated  that  the  follow- 
up responses  would  provide  specific  task-related  informa- 
tion that  would  be  helpful  in  looking  at  tasks  for  which 
other  indices  indicated  there  might  be  a problem. 
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IV  RESULTS  OF  TRIAL  SURVEY 

The  data  in  this  section  are  those  collected  in 
regard  to  Course  3ABR  42330.  They  are  presented  primarily 
to  demonstrate  the  type  of  data  that  were  collected  and 
analysed  rather  than  to  provide  a substantive  evaluation 
of  this  course.  Most  of  the  interpretations  included  are 
general  in  nature.  They  will  necessarily  reflect  the 
specifics  of  the  demonstration  course,  but  comments  about 
generalizations  to  other  courses  and  ether  specialties 
will  be  included. 

Description  of  the  Evaluation  Sample 

The  evaluation  samp le  included  in  the  study  repre- 
sented five  separate  classes  from  the  technical  training 
course.  Selected  from  the  students  in  these  five  classes 
were  186  graduates.  Field  evaluation  questionnaires 
were  sent  to  each  of  these  186  graduates  and  their  super- 
visors. Ideally,  an  evaluation  sample  should  be  somewhat 
larger  than  this  but  trade-offs  between  time  required 
to  aggregate  a larger  sample  and  the  increased  reliability 
and  validity  that  such  samples  might  provide  were  made. 
Throughout  the  Air  Force,  student  flow  in  any  particular 
course  will,  of  course,  determine  feasible  evaluation  samples. 

Response  rate: 

Returns  from  the  questionnaire  mailed  to  the  field 
are  shown  in  Table  1.  Overall,  responses  were  received 


Table  1 

Questionnaire  Response  Rate 


Return  Status 

Supervisor 

Questionnaires 

Graduate 

Questionnaires 

Matched  Super- 
visor/Graduate 
Questionnaires 

N 

% 

N 

% 

N 

% 

Returned  complete 

122 

66 

130 

70 

95 

51 

Returned  incomplete 

21 

11 

8 

4 

70 

38 

Not  returned 

43 

23 

48 

26 

21 

11 
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from  143  supervisors,  which  represents  77%  of  the  sample. 

Of  these,  122  were  reasonably  complete  and  usable  for 
later  analyses.  These  represented  66%  of  the  sample 
mailed.  Overall  responses  were  received  from  138  graduates. 
Which  represents  74%  of  the  sample.  Of  these,  130  were 
reasonably  complete  and  usable  for  later  analyses.  These 
represented  70%  of  the  sample  mailed.  Each  technical 
training  wing  may  compare  these  return  rates  with  those 
normally  received  during  field  evaluation.  Usable  returns 
were  received  from  both  the  graduate  and  supervisor  for 
approximately  51%  of  the  sample  mailed,  and  nothing  at 
all  was  heard  from  approximately  11%  of  the  sample  to 
which  questionnaires  were  mailed.  The  implications  of 
these  return  rates  were  that  analyses  based  solely  on 
supervisor  data  included  about  two-thirds  of  the  sample, 
those  based  solely  on  graduate  data  included  about  three- 
quarters  of  the  sample,  and  those  based  on  comparisons 
of  paired  cases  were  based  on  about  one-half  of  the  sample. 

Sample  characteristics: 

The  distribution  of  grade  levels  is  shown  in  Table  2 
broken  down  by  major  command.  Approximately  64%  of  the 
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TABLE  2 


Grade  Level  of  Graduate  Responses  by  iVlajor  Command 


Major 

Command 

Missing 

E-2 

Grade  Level 

E-3  E-4 

E-5 

Command  Total 

N % 

AAC 

0 

1 

2 

0 

0 

3 

2 

ADC 

0 

4 

0 

0 

0 

4 

3 

AFSC 

0 

3 

0 

0 

c 

3 

2 

ATC 

0 

2 

1 

0 

0 

3 

2 

MAC 

0 

16 

1 

0 

1 

18 

14 

PACAF 

0 

4 

f 

0 

0 

5 

4 

SAC 

1 

41 

18 

0 

0 

60 

46 

TAC 

0 

7 

18 

1 

1 

27 

21 

USAFA 

0 

0 

1 

0 

0 

1 

1 

USAFE 

0 

4 

2 

0 

0 

6 

5 

Grade  Total: 

n 1 

82 

44 

1 

2 

130 

100 

% i 

64 

34 

1 

2 

100 

100 
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sample  were  at  the  E-2  level,  34%  at  the  E-3  level,  with 
the  remainder  at  E-4  and  E-5.  Command-wise,  46%  were 
from  SAC,  21%  from  TAC,  14%  from  MAC,  with  the  remaining 
19%  being  spread  across  seven  other  major  commands.  For 
this  particular  course,  SAC  is  the  major  user  of  graduates, 
with  TAC  and  MAC  important  secondary  users.  Table  3 shows 
the  distribution  of  duty  assignments  of  the  sample.  The 
bulk  of  the  sample  (83%)  was  Apprentice  Aircraft  Elec- 
trical System  Specialists,  with  most  of  those  remaining 
being  Aircraft  Electrical  System  Specialists. 


TABLE  3 

Graduate  Duty  Assignment  by  Major  Command 


Major 

Command 

Apprentice  Air- 
craft Electrical 
Systems  Spec. 

Aircraft 
Electrical 
Systems 
Special  ist 

Aircraft 

Electrical 

Systems 

Technician 

Other 

AAC 

2 

0 

0 

0 

ADC 

3 

2 

0 

0 

AFLC 

1 

0 

0 

0 

AFSC 

2 

0 

0 

0 

ATC 

2 

1 

0 

0 

MAC 

15 

0 

0 

1 

PACAF 

4 

2 

0 

0 

SAC 

48 

11 

1 

0 

TAC 

17 

3 

0 

0 

USAFA 

1 

0 

0 

0 

USAFE 

6 

0 

0 

0 

All  Commands:  n 101 

19 

1 

1 

% 83 

16 

<1 

<1 

Background  questions: 

Several  of  the  background  questions  involved  the 
amount  of  formal  training  that  had  been  or  was  being  taken 
by  the  graduates  after  completion  of  the  resident  course. 
Tables  4 and  5 show  data  collected  from  graduate  ques- 
tionnaires in  this  area.  Table  6 is  based  on  data  taken 
from  the  supervisor  questionnaires.  Approximately  58% 
of  the  responding  graduate  sample  indicated  that  they 
had  taken  no  formal  training  courses  since  completion 
of  the  resident  course,  32%  reported  taking  one  formal 
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TABLE  4 

Formal  Training  Courses  Graduates  Reported  Taking 
Since  Completion  of  Course  3ABR42330 

Number  of 
of  courses 

Frequency 

Percent  of 
graduates 

None 

76 

58 

1 

41 

32 

2 

8 

6 

3 

5 

4 

( 


TABLE  5 

Time  in  Formal  Training  Courses  Since  Completion 


of  Course  3ABR42330  Reported  by  Graduates 

Weeks 

Frequency 

Percent  of 

Graduates  ' 

None 

76 

58 

1 

7 

5 

2 

9 

7 

3 

11 

8 

4 

7 

5 

5 

3 

2 

6 

4 

3 

7 

1 

1 

8 

1 

1 

9 

2 

2 

12 

4 

3 

15 

1 

1 

16 

2 

2 

18 

1 

1 

26 

1 

1 

Average  weeks  reported  in  formal  training  by  54  of  the  1 30  Graduates  - 5.5 
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course,  and  10%  reported  taking  more  than  one  formal 
course.  In  terms  of  the  amount  of  graduate  time  spent 
in  such  formal  training  courses,  Table  5 indicates  that 
the  reported  range  was  from  1 to  26  weeks,  the  latter 
being  an  anomaly  since  there  theoretically  was  not  that 
much  time  available  within  the  evaluation  time  frame. 

About  20%  of  the  sample  reported  spending  between  1 to 
3 weeks  in  formal  training,  while  the  remainder  was 
scattered  across  the  remaining  range.  The  amount  of 
training  normally  undertaken  by  resident  course  graduates 
immediately  after  graduation  has  an  influence  on  the 
amount  of  time  they  can  actually  spend  on  performing 
job  tasks  within  the  arbitrary  time  frame  in  the  present 
evaluation  process.  Specialties  undoubtedly  vary  in 
this  respect  and  interpretations  from  field  evaluation 
samples  should  take  this  into  consideration.  While  approx- 
imately 3/5  of  the  sample  reported  no  additional  formal 
course  training,  supervisors  reported  that  over  9/10  of 
the  sample  was  enrolled  in  or  had  completed  the  relevant 
CDC,  as  shown  in  Table  6.  Supervisors  also  reported  that 
almost  2/5  of  the  sample  was  or  had  been  involved  in 
formal  FTD  courses. 


TABLE  6 

Graduate  Involvement  in  Additional  Formal  Training 

Based  on  1 22  Supervisor  Reports 

Career  Develop- 
ment Course 

FTD  Course 

Yes 

114 

46 

No 

8 

76 

* > 


The  final  course  grade  was  one  criterion  of  overall 
success  that  was  available  from  records  maintained  at  the 
technical  training  wing.  The  distribution  of  these  grades 
is  shown  in  Table  7.  The  final  grades  ranged  from  61  to 
92  with  both  a mean  and  median  grade  of  77. 
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TABLE  7 

Distribution  of  Final  Course  Grades 

Grade 

Frequency 

Percent 

Cumulative 

Percent 

70  and  below 

21 

ii 

ii 

71 

10 

5 

16 

72 

10 

5 

21 

73 

7 

4 

25 

74 

15 

8 

33 

75 

16 

9 

42 

76 

4 

2 

44 

77 

14 

8 

52 

78 

16 

9 

61 

79 

10 

5 

66 

80 

12 

6 

72 

81 

8 

4 

76 

82 

3 

2 

78 

83 

3 

2 

80 

84 

13 

7 

87 

85  and  above 

24 

13 

100 

N = 186;  Mean  = 77.45; 

S.D.  = 5.97;  Range  = 61-92 

Since  there  was  an  interest  in  the  capabilities  of 
the  female  personnel  assigned  to  this  specialty  in  hand  ling 
the  physical  aspects  involved,  a specific  question  had 
been  included  in  the  supervisor  questionnaire  by  the 
Training  Evaluation  Division.  There  were  36  female  gradu- 
ates included  in  the  overall  evaluation  sample  of  186. 

Of  the  122  supervisors  reporting,  16  (13%)  gave  no  response, 
which  theoretically  indicated  that  no  females  were  assigned 
to  their  work  site.  Of  the  remaining  supervisors,  66 
(54%)  indicated  that  females  were  "physically  capable  of 
performing  all  of  the  tasks  in  this  AFSC  that  the  male 
personnel  normally  perform,"  and  40  (33%)  indicated  that 
they  were  not.  These  data  are  presented  in  Table  8. 
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TABLE  8 

Physical  Capability  of  Females  to  do  the  Job 


Response  to : Are  the  female  personnel  assigned  to  your  shop  physically 
capable  of  performing  all  of  the  tasks  in  the  AFSC  that 
the  male  personnel  normally  perform? 


Frequency 

N 

% 

No  response  or 

no  females  assigned 

16 

13 

Yes 

66 

54 

No 

40 

33 

Ability  Indices: 

The  ability  levels  of  the  evaluation  sample  as  mea- 
sured by  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  and  the  Armed  Forces  Qualification  Test  (AFQT) 
composite  scores  derived  from  this  battery,  are  shown 
in  Table  9.  The  distributions  shown  reflect  the  combined 
effects  of  the  cut-off  score  of  50  on  the  Electronic  Apti- 
tude Index  for  entrance  into  the  course  and  the  inter- 
correlation of  this  index  with  the  other  ability  scores. 
Inspection  of  the  table  reveals  the  rather  severe  re- 
striction in  range  of  scores  on  the  Electronic  and 
General  Aptitude  indices  and  on  the  AFQT  Composite. 

Similar  patterns  will  probably  be  found  for  other  spe- 
cialties with  only  the  controlling  aptitude  indices 
changing  to  reflect  the  basic  nature  of  the  specialty 
involved . 

In  an  effort  to  determine  interaction  effects  between 
the  ability  scores  and  several  different  general  criterion 
measures,  a correlation  matrix  was  developed.  This  is 
shown  in  Table  10.  Three  different  criterion  measures 
were  used.  One  was  final  course  grade.  Another  was  av- 
erage rated  proficiency.  This  index  was  derived  for  any 
given  individual  by  summing  the  performance  ratings  given 
by  supervisors  on  each  task  performed  and  dividing  by  the 
number  of  tasks  performed.  The  third  criterion  used  was 
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TABLE  9 


Distribution  of  ASVAB  Ability  Scores  of  Graduate  Sample 


Score 

AFQT 

Composite 

Administrative 

Aptitude 

Electronic 

Aptitude 

General 

Aptitude 

Mechanical 

Aptitude 

N 

% 

N 

% 

N 

% 

N 

% 

N 

% 

0-10 

0 

0 

2 

1 

0 

0 

0 

0 

6 

4 3 

11-20 

0 

0 

1 

1 

0 

0 

0 

0 

8 

4 

21-30 

0 

0 

12 

7 

0 

0 

0 

0 

26 

15 

31-40 

1 

1 

20 

11 

0 

0 

0 

0 

27 

15 

41-50 

34 

20 

19 

11 

6 

3 

20 

11 

17 

9 

51-60 

29 

17 

37 

21 

32 

18 

38 

21 

17 

9 

61-70 

45 

26 

31 

17 

60 

34 

33 

18 

22 

12 

71-80 

36 

21 

23 

13 

41 

23 

34 

19 

18 

10 

81-90 

14 

8 

22 

12 

33 

18 

41 

23 

29 

16 

91-99 

12 

7 

11 

16 

6 

3 

1 2 

7 

8 

4 

n 

171 

178 

178 

178 

178 

Mean 

65.80 

61.72 

72.02 

71.40 

55.08 

S.D. 

14.44 

20.54 

11.67 

11.6? 

25.39 

Range 

39-97 

1-95 

50-95 

45-95 

5-95 

the  supervisor 1 s , rating  to  the  question  on  "Graduate's 
Overall  Ability  to  Perform  Duties." 

A review  of  the  matrix  indicates  that  all  of  the  abil- 
ity measures  are  significantly  intercorrelated  with  the 
one  exception  being  the  Mechanical  and  Administrative  Apti- 
tude indices.  Similar  results  would  undoubtedly  be  found 
in  examining  a similar  matrix  for  any  other  specialty.  The 
matrix  also  shows  that  all  of  the  ability  scores  are  posi- 
tively correlated  with  final  course  grade.  This  same 
result  has  been  repeatedly  found  in  past  studies  on  any 
number  of  different  specialties.  The  ability  scores  used 
by  the  Air  Force  routinely  predict  success  in  training  as 
measured  by  course  grades.  Unfortunately,  the  matrix 
also  shows  no  significant  re  1 a t i onships  between  any  of  the 
ability  measures  and  job  performance  as  measured  by  aver- 
age rated  proficiency.  The  situation  is  much  the  same 
when  the  rated  overall  ability  to  perform  duties  is  the 
criterion  of  job  performance.  In  this  case,  however, 
measures  of  General  Aptitude  did  show  a modest  but 
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TABLE  10 

Intercorrelations  between  Ability  Indices  and  Indices  of  General  Proficiency 

Administra- 
tive Aptitude 

Electronic 

Aptitude 

General 

Aptitude 

Mechanical 

Aptitude 

Final 

Grade 

Average 

Proficiency 

Overall 

Ability 

AFQT  Composite 

41  ** 

62** 

74** 

31** 

33** 

13 

15 

Administrative 

Aptitude 

20** 

53** 

9 

26** 

2 

10 

Electronic 

Aptitude 

43** 

51** 

37** 

2 

2 

General  Aptitude 

24** 

33** 

13 

22* 

Mechanical 

Aptitude 

15* 

7 

13 

Final  Course 

Grade 

7 

11 

Average 

Proficiency 

59** 

* Statistically  significant  at  5%  level;  **  Statistically  significant  at  1%  level. 
N's  vary  between  112-178;  decimal  points  have  been  omitted. 


significant  relationship  with  the  criterion.  This  general 
result  of  ability  to  predict  course  grades  but  not  on- 
the-job  performance  is  also  in  line  with  results  obtained 
in  many  previous  studies.  An  interesting  finding  shown  on 
the  matrix  is  the  significant  positive  relationship  between 
the  two  job  performance  criterion  measures.  Because  of 
the  format  of  the  questionnaire,  most  supervisors  would 
have  made  and  recorded  their  judgment  on  overall  ability 
to  perform  duties  before  any  consideration  of  the  81  tasks. 
Consideration  of  task  details  should  therefore  not  have 
influenced  these  general  ratings. 

Tables  11  and  B-l  through  B-4  in  Appendix  B display 
the  relationship  between  ability  measures  and  the  overall 
ability  to  perform  measures  in  a somewhat  different 
fashion.  The  tabular  formatting  of  these  tables  allows 
one  to  directly  address  the  question  of  the  potential 


TABLE  11 

Overall  Ability  to  Perform  versus  Electronic  Aptitude 


Electronic  Aptitude 
Score 

Missing 

Unsatis- 

factory 

Marginal 

Satisfactory 

Very  satis- 
factory 

Excellent 

91-100 

2 

0 

i 

0 

1 

2 

81-90 

11 

0 

3 

11 

5 

3 

71-80 

14 

0 

2 

16 

5 

4 

61-70 

18 

1 

5 

21 

9 

6 

51-60 

13 

0 

0 

10 

5 

4 

41-50 

2 

0 

0 

3 

1 

0 

Missing 

5 

0 

1 

2 

0 

0 

Percent  of  Response 
<n=1 18) 

1 

9 

52 

22 

16 

* 
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effects  of  adjusting  ability  cut-off  scores  on  the 
rated  overall  ability  to  perform.  By  drawing  a horizontal 
line  across  the  table  at  any  designated  or  hypothesized 
ability  cut-off  level  and  then  running  a perpendicular 
line  between  any  two  levels  of  the  criterion  that  one 
wishes  to  consider,  the  number  in  each  quadrant  will 
immediately  indicate  the  number  of  "good  ones  and  bad 
ones"  lost  and  "good. ones  and  bad  ones"  retained  by  any 
hypothesized  cut-off  level.  Such  an  analysis  performed 
on  the  data  from  the  demonstration  course  indicated  that 
no  possible  alternative  cut-off  scores  on  one  or  more  of 
the  ability  indices  could  be  instituted  without  such  action 
dropping  more  satisfactorily  performing  graduates  than  it 
eliminated  unsatisfactorily  performing  graduates.  Within 
the  constraints  of  abilities  as  measured  by  the  ASVAB 
(and  AFQT  composite)  and  job  performance  as  measured  by 
supervisor  ratings  of  overall  ability,  manipulation  of 
graduate  input  on  these  variables  is  not  a warranted 
action.  These  data  cannot,  however,  indicate  what  would 
happen  if  the  gateway  aptitude  index  (in  this  study 
Electronics  at  50)  were  dropped.  The  same  type  Qf  anal- 
yses could,  of  course,  be  run  against  other  criteria  of 
job  performance.  So  long  as  one  accepts  the  validity  of 
the  criterion  measure  used,  this  type  of  analysis  pro- 
vides a relatively  simple  forecast  of  what  would  happen 
if  ability  measures  were  raised,  and  it  can  be  applied 
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across  all  specialties.  Over  a period  of  time  it  would 
be  possible  to  successively  lower  the  input  requirements 
in  a specialty  or  course  until  a point  was  reached  where 
the  cut-off  allowed  more  unsatisfactory  performers  into 
the  specialty  than  satisfactory  ones. 

General  Reactions  to  the  Survey  Questionnaires 

At  the  end  of  each  survey  questionnaire,  with  the 
exception  of  the  diagnostic  follow-up  questionnaires, 
respondents  were  asked  about  their  overall  reactions  to 
the  instrument.  Results  of  the  responses  to  these  ques- 
tions are  shown  on  Tables  B-5  through  B-19  in  Appendix  B. 
Reactions  to  the  basic  supervisor  questionnaire  indicated 
that  both  the  instructions  and  scales  were  clear  to  the 
vast  majority  of  respondents.  About  one-third  of  the 
supervisors  took  30  minutes  or  less  to  complete  the  ques- 
tionnaire, another  third  took  between  30  to  45  minutes, 
and  the  remaining  third  took  longer.  There  was  no  clear 
preference  for  responding  on  the  questionnaire  itself  or 
on  separate  answer  sheets  although  there  was  a slight  pref- 
erence for  the  latter.  It  is  interesting  to  note  that  only 
31%  of  the  supervisors  reported  that  they  had  previously 
completed  a field  evaluation  questionnaire. 


Reactions  to  the  basic  graduate  questionnaire  indi- 
cated that  the  instructions  and  scales  were  clear  to  the 
majority  of  the  respondents.  About  two-fifths  of  the 
sample  took  20  minutes  or  less  to  complete  it,  another  one- 
third  took  between  25  and  35  minutes,  and  the  remainder 
took  longer.  There  was  no  clear  preference  for  responding 
on  the  questionnaire  itself  or  on  separate  answer  sheets 
although  there  was  a slight  preference  for  the  latter. 

Reactions  to  Supplemental  Questionnaire  A - Task 
Criticality  indicated  that  the  instructions  and  scales 
were  clear  to  the  majority  of  the  respondents.  Slightly 
less  than  one-third  of  the  respondents  took  15  minutes  or 
less  to  complete  it,  about  one-half  took  between  20  and  30 
minutes,  and  the  remainder  took  longer.  The  vast  majority 
of  the  respondents  felt  comfortable  in  making  the  ratings 
and  felt  they  had  provided  valid  data. 

Reactions  to  Supplemental  Questionnaire  B - Training 
Level  Standards  indicated  that  the  instructions  and  scales 
were  clear  to  the  majority  of  the  respondents.  About  two- 
fifths  of  the  respondents  took  15  minutes  or  less  to  com- 
plete it,  about  one-third  took  between  20  to  30  minutes, 
and  the  remainder  took  longer.  Over  three-quarters  of 
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the  respondents  felt  comfortable  in  making  these  ratings 
and  felt  they  had  provided  valid  data.  About  one-eighth 
of  the  sample  felt  a little  uneasy  about  doing  the  task 
but  felt  they  had  provided  valid  data.  About  one-eighth 
of  the  sample  felt  uneasy  about  making  such  ratings  and 
questioned  the  validity  of  the  data  they  provided. 

Reactions  to  Supplemental  Questionnaire  C - Impact 
of  Training  Deficiency  indicated  that  the  instructions 
and  scales  were  clear  to  the  majority  of  the  respondents. 
About  one-half  of  the  respondents  took  25  minutes  or  less 
to  complete  it,  about  two-fifths  took  30  minutes,  and  the 
remainder  took  longer.  Almost  nine-tenths  of  the  respond- 
ents felt  comfortable  about  making  these  ratings  and  felt 
they  had  provided  valid  data.  The  remaining  one-tenth 
were  a little  uneasy  about  making  such  ratings. 

Twenty-seven  of  the  122  supervisors  (22%)  used 
the  opportunity  to  respond  meaningfully  to  the  open-ended 
comment  section.  Thirty-nine  of  the  130  graduates  (30%) 
also  provided  open-ended  comments. 
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Reactions  to  the  questionnaires  in  general  indicated 
that  even  though  the  bulk  of  the  sample  had  not  responded 
to  field  evaluation  questionnaires  before,  and  certainly 
none  to  the  special  forms  used  in  this  study,  the  instruc- 
tions were  generally  well  understood.  Average  response 
times  per  questionnaire  were  between  26  and  47  minutes 
which  would  appear  to  be  well  within  an  acceptable  range. 
There  was  a slight  but  not  strong  preference  for  the  use 
of  separate  answer  sheets  for  the  basic  supervisor  and 
graduate  questionnaires.  The  majority  of  the  supervisor 
respondents  felt  comfortable  in  making  the  types  of  ratings 
requested  on  the  supplemental  questionnaires  and  felt  they 
had  provided  valid  data  on  them.  From  the  standpoint  of 
user  acceptability,  therefore,  all  of  the  instruments 
used  in  the  present  study. appear  to  be  feasible  for  Air 
Force-wide  use. 


Analyses  by  Job  Clusters 

For  the  present  study,  the  job  clusters  used  were 
originally  derived  from  data  received  from  experienced 


work  center  supervisors  which  were  translated  into  a 
question  with  four  options  to  represent  four  somewhat 
different  job  clusters  within  the  specialty.  For  any 
specialty  for  which  there  were  recent  Occupational  Survey 


data,  empirically  determined  job  clusters  could  be  used 


as  well.  The  same  concept  could  also  be  applied  to 
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different  types  of  hardware  and/or  weapon  systems  if  they 
were  of  particular  interest  in  any  given  specialty.  Job 
clusters,  or  hardware  or  weapon  system  clusters,  or  any 
other  relevant  dimension  could  be  used  as  a control  vari- 
able with  which  to  divide  the  total  specialty  sample  into 
meaningful  subgroups  for  comparative  analyses.  Four  job 
clusters  were  utilized  in  the  present  study.  They  were 
identified  by  both  graduates  and  by  supervisors  in  the 
field  survey  through  answers  to  the  same  four-option  ques- 
tion. Supervisors  were  forced  to  choose  only  one  job 
cluster.  The  job  clusters  reported  by  graduates  and  super- 
visors are  shown  in  Table  12  broken  down  by  major  command. 
Almost  one-half  of  the  sample  was  in  Cluster  B,  about  one- 
quarter  in  Cluster  A,  and  the  remainder  somewhat  evenly 
split  between  Clusters  C and  D.  Both  Clusters  C and  D 
were  shop  oriented,  but  Cluster  D specifically  included 
the  troubleshooting  component  while  Cluster  C did  not. 
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TABLE  12 

Job  Tasks  on  Which  Graduates  Spent  Most  Time  by  Major  Command 


Major  Command 

Group  A* 

Super- 

Graduate  visor 
Report  Report 

Group  B* 
Super- 

Graduate  visor 
Report  Report 

Group  C* 

Super- 

Graduate  visor 
Report  Report 

Group  D* 

Super- 

Graduate  visor 
Report  Report 

AAC 

3 

1 

1 

1 

0 

0 

0 

0 

ADC 

1 

2 

2 

3 

2 

0 

2 

0 

AFLC 

0 

0 

0 

1 

0 

0 

0 

0 

AFSC 

0 

0 

2 

2 

0 

0 

1 

0 

ATC 

0 

0 

2 

2 

1 

1 

0 

0 

MAC 

5 

2 

10 

7 

3 

4 

6 

3 

PACAF 

0 

0 

1 

3 

2 

2 

2 

1 

SAC 

23 

16 

30 

29 

2 

7 

12 

8 

TAC 

4 

2 

15 

14 

5 

1 

4 

3 

U3AFA 

1 

1 

0 

0 

0 

0 

0 

0 

USAFE 

0 

2 

4 

4 

0 

0 

1 

0 

Group  Total: 

N“  37 

26 

67 

66 

15 

15 

28 

15 

%*~  25 

21 

46 

54 

10 

12 

29 

12 

• Group  A * Battery  maintenance  functions 

Group  B * Visually  inspecting  and  troubleshooting  electrical  circuits  and  systems  on  aircraft. 
Group  C * Performing  general  shop  and  bench  checking  tasks  such  as  assembly,  disassembly, 
and  internal  repair  tasks. 

Group  D - Troubleshooting  and  general  shop  functions 
” 12  of  the  130  graduates  reported  spending  equal  time  in  more  than  one  group. 

Percent  for  graduates  based  on  147  total  responses,  including  the  multiple  responses.  Percent 
for  supervisors  based  on  1 22  responses. 
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The  designation  of  job  cluster  was  based  on  responses 
to  a question  phrased  in  terms  of  the  activities  in  which 
the  graduate  spent  most  of  his  time  since  completion  of 
resident  training.  For  cases  on  which  both  graduate  and 
supervisor  responses  had  been  received,  it  was  possible 
to  check  the  agreement  of  this  "most  time  spent"  desig- 
nation. Table  13  summarizes  the  amount  of  agreement 
between  graduate  and  supervisor  responses  on  this  question. 
Best,  though  not  total  agreement,  is  shown  for  the  cluster 
concerning  troubleshooting  aircraft.  The  battery  main- 
tenance cluster  is  next  best,  with  more  agreement  than 
disagreement.  For  both  of  the  remaining  clusters,  there 
was  more  disagreement  than  agreement  on  where  most  time 
was  spent.  Part  of  the  disagreements  are  artifacts  caused 
by  forcing  supervisors  to  a single  rating  category  and 
allowing  graduates  multiple  ratings.  In  any  future  uses 
of  this  approach,  the  question  response  options  and  rules 
must  be  identical  for  both  supervisors  and  graduates  so 
that  direct  comparisons  can  be  made  without  contamination 
from  the  multiple  response  option. 


TABLE  13 
Reported  J ob  T ypes 


Graduate  Reports  of 

Most  Time  Spent 

a .ff  ra  i - ....  ■■■  ■ a—  i — 

Supervisor  Reports  of  Most  Time  Spent 

All  Graduates 

N % 

Battery 

Mainte- 

nance 

Trouble- 

shooting 

Aircraft 

Trouble- 
Checking  shooting 
in  Shop  in  Shop 

Battery  Maintenance 

© 

7 

5 

3 

27 

25 

Troubleshooting  Aircraft 

3 

© 

1 

5 

50 

45 

Checking  in  Shop 

1 ’ 

3 

© 

1 

11 

10 

Troubleshooting  in  Shop 

3 

8 

3 

© 

22 

20 

All  Supervisors: 

N 

19 

59 

15 

17 

110 

% 

17 

54 

14 

15 

Note:  Supervisors  were  forced  to  designate  only  one  job  type  while  graduates  could  report  multiples 
if  they  spent  significant  equal  amounts  of  time  in  more  than  one  job  type.  Agreements  shown 
by  circled  entries  indicate  graduate  reported  spending  a significant  amount  of  time  in  job  type 
and  supervisor  indicated  most  time. 
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One  analysis  made  by  job  clusters,  and  the  same  type 
of  sub-group  analysis  could  be  made  using  other  control 
variables,  was  to  examine  the  extent  to  which  each  of 
the  81  job  tasks  was  primarily  associated  with  any  of  the 
job  clusters.  Data  regarding  this  are  shown  in  Table 
B-20  in  Appendix  B.  All  of  the  cell  entries  in  this 
table  represent  percent  of  the  total  sample.  The  per- 
cent of  the  sample  in  each  job  cluster  is  shown  in  the 
column  heading.  The  extent  to  which  a given  task  is 
associated  with  the  job  cluster  is  determined  by  the 
amount  the  job  task  percentage  is  greater  than  or  less 
than  the  appropriate  column  heading  percentage.  On  those 
tasks  where  the  task  entry  exceeds  the  column  heading, 
the  task  tends  to  be  associated  with  that  job  cluster. 

The  greater  the  discrepancy  upward,  the  stronger  the 
association  with  that  job  cluster.  On  those  tasks  where 
the  entry  is  less  than  the  column  heading,  the  tasks 
tend  not  to  be  associated  with  that  job  cluster.  Again, 
the  greater  the  discrepancy  the  higher  the  degree  of 
disassociation . For  convenience  sake,  the  percent  of 
the  total  sample  reporting  that  the  task  was  performed 
is  also  shown  on  Table  B-20.  It  can  be  noted  that  the 
range  between  tasks  is  from  almost  none  to  almost  all. 

When  examining  other  indices  of  training  adequacy  or  job 
proficiency  at  the  task  level,  the  percent  performing 
can  greatly  influence  whether  or  not  something  should  be 
done  in  regard  to  a specific  task. 

Another  analysis  that  was  carried  out  by  job  clus- 
ters involved  the  overall  criterion  of  average  profi- 
ciency. The  distribution  of  average  proficiency  ratings 
is  shown  in  Table  14  broken  down  by  both  job  cluster  and 
major  command . The  mean  average  rated  proficiency  for 
tasks  in  three  of  the  job  clusters  was  very  close.  The 
mean  for  the  battery  maintenance  cluster  was  somewhat 
higher.  This  could  be  attributed  to  training  variables 
or  variables  associated  with  the  difficulty  level  of  tasks 
involved  with  the  battery  maintenance  function.  These 
types  of  survey  data  will  not  make  that  type  of  causal 
attribution.  There  was  some  spread  on  a command-wide 
basis  from  just  below  the  level  indicating  "Partially 
Competent"  to  well  alcng  the  way  toward  "Competent . " 

An  analysis  of  variance  was  carried  out  using  the  command 
average  proficiency  score  data.  This  analysis  indicated 
that  the  command  differences  observed  in  the  study  were  not 
statistically  significant  (Fg.gg  = 1.0248). 

Data  analyses  of  this  type,  provide  gross  indications 
of  the  degree  to  which  the  graduates  from  the  resident 
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TABLE  14 

Mean  Average  Proficiency  by  Major  Command  and  Job  Type 


Major 

Command 

Job  Type 

Command 

Average 

Battery 

Maintenance 

Troubleshoot- 
ing Aircraft 

Checking 
in  Shop 

Troubleshoot- 
ing in  Shop 

AAC 

2.50 

2.31 

— 

— 

2.40 

ADC 

2.40 

2.80 

- 

- 

2.24 

AFSC 

- 

2.33 

- 

- 

2.33 

ATC 

- 

2.39 

3.17 

- 

2.99 

MAC 

3.23 

2.66 

2.54 

2.45 

2.68 

PACAF 

- 

2.70 

2.34 

- 

2.52 

SAC 

2.91 

2.65 

2.73 

2.58 

2.71 

TAC 

2.59 

2.66 

2.81 

2.91 

2.70 

USAFA 

2.67 

' ■ - 

- 

- 

2.67 

USAFE 

2.66 

2.38 

- 

- 

2.84 

N 

22 

59 

15 

13 

Mean 

2.74 

2.62 

2.66 

2.63 

S.D. 

.60 

.54 

.60 

.66 

Range 

1 .34-4.00 

1.38-3.90 

1.55-4.00 

1.28-4.00 

course  are  meeting  the  differential  needs  represented  by 
command  and  job  cluster  variables.  If  relative  deficien- 
cies are  noted  for  any  specific  type  of  user,  consider- 
ation may  be  given  either  to  trying  to  correct  the  sit- 
uation at  the  resident  course  or  to  specifically  require 
remedial  training  actions  at  the  work  site. 

Reported  Difficulty  On  The  Job  Or  In  OJT 

A relevant  item  of  concern  in  an  evaluation  of  a 
resident  course  is  whether  or  not  graduates  experienced 
any  difficulties  on  their  duty  assignments  which  were 
attributable  to  inadequacies  in  the  resident  course 
training.  Supervisor  field  evaluation  questionnaires 
include  the  question  "Did  the  graduate  experience  any 
difficulty  in  performing  OJT  duties  that  were  due  to 
inadequate  training  in  the  resident  course?"  In  this 
study,  a similar  question  was  included  in  the  graduate 
questionnaire  but  it  was  framed  in  terms  of  "on  the  job 
or  in  OJT  activities."  The  number  of  graduates  report- 
ing having  difficulty  attributable  to  resident  training 
course  inadequacies  is  shown  in  Table  15. 


TABLE  15 

Difficulty  on  Job  or  OJT  Reported  by  130  Graduates 


Number 

Reporting 

Percent 
of  Relevant 
Sample 

Reported  difficulty: 

Female 

1 

5 

Male 

14 

13 

Both  sexes 

15 

12 

Reported  no  difficulty: 

Female 

19 

95 

Male 

96 

87 

Both  sexes 

115 

88 

All  graduates. 

Female 

20 

15 

Male 

110 

85 

Both  sexes 

130 

100 

Data  for  cases  on  which  both  supervisor  and  graduate 
questionnaires  were  available  are  shown  in  Table  16. 

Only  15  cases  representing  12  percent  of  the  evaluation 
sample  reported  having  any  difficulties.  Only  one  female 
reported  having  difficulty.  The  data  in  Table  16  shows 
the  amount  of  agreement  between  self  reports  of  difficulty 
and  supervisor  reports  of  difficulty.  If  one  looks  at 
the  data  for  all  job  types,  it  is  apparent  that  the  largest 
area  of  agreement  is  when  both  the  graduate  and  super- 
visor had  not  perceived  any  difficulties.  While  the 
agreement  is  high  for  both  sexes,  it  is  somewhat  higher 
for  male  graduates  than  for  female  graduates.  For  the 
ten  cases  in  this  sample  of  matched  cases  in  which  the 
graduate  reported  a difficulty,  in  only  one  case  had  the 
supervisor  perceived  any  difficulty,  which  represents  only 
a 10%  agreement.  If  the  results  of  the  field  evaluation 
for  this  study  are  representative  of  what  usually  occurs 
in  the  present  system,  the  use  of  supervisor  reports  only 
may  be  understating  the  number  of  cases  in  which  diffi- 
culties are  being  experienced.  The  value  judgment  as  to 
which  report,  the  graduate's  or  the  supervisor's,  is  the 
more  valid  cannot  be  made  from  the  survey  data  themselves. 
Even  if  the  supervisor's  judgments  are  deemed  more  valid 
because  they  presumably  represent  the  acceptable  criterion 
level,  any  sizable  amount  of  graduate  perceived  difficul- 
ties which  are  at  variance  with  that  criterion  might  have 
a deleterious  effect  on  the  motivational  level  of  the 


TABLE  16 

Reported  Difficulty  on  Job  or  OJT  on  96  Cases  for  Which  Both 
Graduate  and  Supervisor  Reports  were  Available 


Graduate  Reported  Difficulty  on  Job  or  OJT 


Difficulty  on  OJT  by 

Job  Type* 

Yes 

No 

Male 

Female 

Both  Sexes 

Male 

Female 

Both  Sexes 

Battery  maintenance: 

Yes 

0 

0 

0 

1 

0 

1 

No 

5 

0 

5 

© 

0 

© 

Troubleshooting  aircraft: 

Yes 

0 

0 

0 

3 

1 

4 

No 

1 

0 

1 

© 

© 

© 

Checking  in  shop: 

Yes 

0 

0 

0 

1 

0 

1 

No 

0 

0 

0 

Q 

© 

© 

Troubleshooting  in  shop: 

Yes 

0 

0 

0 

0 

2 

2 

No 

2 

1 

3 

© 

© 

© 

All  Job  Types. 

Yes 

© 

0 

© 

5 

3 

8 

No 

8 

1 

9 

© 

© 

© 

All  supervisor  ratings: 

9 

0 

10 

73 

13 

86 

Percent  agreement: 

11 

0 

10 

93 

77 

91 

• Job  type  determined  by  graduate  report  of  where  she/he  spent  most  time. 
Circled  entries  indicate  cases  of  agreement. 


graduate . 
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Id  probably  also  me 
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Grad 

uate 
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in 
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the  graduate,  CDC 
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indicate  a "not  app 

licable"  response  if 

the  t 
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been  performed  or  w 
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rk  . 
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argument 
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were  requested  only  for  those  tasks  in  which  graduates 
had  participated  and  therefore  did  have  an  experiential 
base  for  making  the  judgments.  The  graduate  reports  of 
training  adequacy  by  job  task  are  shown  in  Table  B-21  in 
Appendix  B. 

The  type  of  analyses  that  can  readily  be  made  from 
this  type  of  data  will  be  illustrated  by  using  the  data 
from  the  Task  Performance  column.  Similar  analyses  can, 
of  course,  be  made  for  the  CDC  and  FTD  columns. 

Examination  of  Table  B-21  indicates  that  the  modal 
response  for  80  of  the  81  job  tasks  was  the  Adequate 
Response.  For  one  of  the  81  tasks,  as  many  graduates 
used  the  Inadequate  response  as  the  Adequate  response. 
Overall,  this  seems  to  indicate  a healthy  picture  for  this 
particular  course  and  no  drastic  changes  need  be  contem- 
plated on  the  basis  of  graduate  perceptions  of  training 
adequacy.  However,  if  task  performance  indices  indicated 
potential  deficiencies  or  if  arbitrary  cuts  in  training 
resources  were  mandated  by  external  circumstances,  it 
may  be  necessary  to  take  a closer  look  at  the  task  data. 
This  may  be  accomplished  by  comparing  the  number  of 
Inadequate  responses  and  Overadequate  responses.  For 
this  particular  course,  53  of  the  81  tasks  had  more  Inad- 
equate responses  than  Overadequate  responses,  and  they 
were  mostly  concentrated  in  different  STS  sections  which 
are  translatable  to  course  blocks.  Onlv  26  of  the  tasks 
had  more  Overadequate  responses  than  Inadequate  responses, 
and  these  too  were  concentrated  in  different  STS  sections. 
The  task  population  available  for  potential  reduction 
based  on  this  index  of  overtraining  is  only  half  the  size 
of  the  task  population  which  could  perhaps  stand  addi- 
tional, or  more  effective,  training.  There  were  equal 
numbers  of  Inadequate  and  Overadequate  responses  for  two 
of  the  81  job  tasks.  In  summary,  all  tasks  appear  to  be 
adequately  trained,  but  approximately  two-thirds  of  them 
are  adequate  leaning  towards  inadequate  while  only  about 
one-third  are  adequate  leaning  towards  overadequacy.  Such 
analyses  also  provide  one  measure  of  overall  task  train- 
ing adequacy.  The  calculated  task  averages  shown  in 
Table  B-21  provide  another  measure  of  overall  task  train- 
ing adequacy. 

Supervisor  Reports  of  Job  Performance 

Supervisor  reports  of  the  demonstrated  proficiency 
level  of  recent  graduates  on  the  job  tasks  relevant  to 
any  given  resident  course  have  been  the  crux  of  the 
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present  field  evaluation  system.  They  have  provided  the 
primary  criterion  from  which  to  make  the  initial  judgment 
of  whether  or  not  the  tasks  were  being  adequately  trained. 
The  evaluation  concept  is  quite  simple.  The  STS  sets  the 
level  of  proficiency  to  be  obtained  as  a training  require- 
ment, and  supervisor  ratings  of  performance  using  cate- 
gories roughly  translatable  to  the  levels  stated  in  the 
training  requirements  provide  the  criterion  of  attainment. 
Percent  attainment  is  then  utilized  as  an  index  of  train- 
ing adequacy  based  on  performance  on  the  job.  Over  the 
years,  guidelines  have  been  provided  as  to  what  percent 
attainment  should  be  used  for  making  a judgment  that 
remedial  action  at  the  resident  course  level  was  necessary. 
The  80  percent  criterion  has  usually  been  applied,  but 
unfortunately,  in  some  cases  it  has  been  used  slavishly 
as  a rule  rather  than  the  guideline  it  was  intended  to  be. 
The  percent  attainment  is  an  index  of  how  well  recent 
graduates  are  judged  to  be  meeting  the  proficiency  level 
expected  and  as  such  is  useful  in  identifying  tasks  on 
which  significant  numbers  of  graduates  are  not  performing 
up  to  expectations.  The  index,  however,  can  only  identify 
the  tasks  but  it  cannot  provide  information  as  to  the 
cause  of  the  deficiency  nor  the  most  effective  locus  for 
remedial  action.  A thorough  examination  of  all  of  the  fac- 
tors surrounding  both  the  training  and  work-site  factors 
associated  with  troublesome  tasks  should  be  made  before 
recommendations  for  changes  are  made.  This  can  only  be 
done  by  personnel  who  are  thoroughly  familiar  with  the 
specialty  involved. 

A summary  of  the  performance  ratings  provided  by  the 
supervisors  in  this  study  is  presented  in  Table  B-22  in 
Appendix  B.  For  each  of  the  81  job  tasks  included  on  the 
supervisor  questionnaire,  the  table  includes  the  number 
and  percent  of  the  sample  of  returned  cases  who  had  not 
performed  the  task,  the  frequency  of  response  in  each  of 
the  proficiency  levels,  the  percent  of  the  sample  con- 
sidered meeting  expectations  using  the  Partially  Proficient 
level  as  a criterion  for  the  lower  limit  of  acceptance, 
and  the  mean  proficiency  rating  for  the  task. 

If  one  uses  the  80%  criterion  for  a guideline,  it 
is  apparent  that  21  of  the  81  job  tasks  do  not  meet  this 
criterion.  This  looks  as  if  about  one-quarter  of  the 
tasks  show  a performance  deficiency  and  that  the  course 
is  in  trouble.  Further  examination,  however,  reveals 
that  all  is  not  that  bad.  To  begin  with,  the  job  tasks 
listed  on  the  supervisor  and  graduate  questionnaires 


exceeded  the  number  of  STS  items.  For  example,  an  STS 
item  stated  as  "use  and  care  of"  was  reflected  as  two  sep- 
arate items  on  the  supervisor  and  graduate  questionnaire 
task  list.  Similarly,  any  STS  item  involving  "operate, 
inspect,  and  troubleshoot"  was  treated  as  three  separate 
job  tasks.  Examination  of  the  job  tasks  not  meeting  the 
80%  guideline  indicate  that  only  12  STS  items  are  involved. 
For  three  of  these  items,  the  level  of  proficiency  stated 
as  a training  and  performance  requirement  was  lower  than 
the  usual  2b  level  which  roughly  translates  to  the  Partially 
Proficient  level  shown  in  Table  B-22.  Of  the  remaining 
nine  STS  items,  all  had  substantial  percentages  of  the 
graduates  not  performing  the  task.  Two  were  in  excess  of 
80%,  two  in  the  70's,  two  in  the  60's,  and  three  between 
48%  and  58%.  Thus,  while  performance  deficiencies  existed 
on  these  tasks,  the  effect  of  these  deficiencies  was  di- 
minished somewhat  by  the  fact  that  half  or  more  of  the 
graduates  were  not  performing  the  tasks.  If  the  graduates 
simply  had  not  been  on  the  job  long  enough  to  get  around 
to  doing  the  tasks,  but  probably  would  before  the  end  of 
their  first  assignment,  then  the  importance  of  the  defi- 
ciencies would  not  be  appreciably  diminished. 


Data  regarding  tasks  performed  and  not  performed  are 
shown  in  Tables  B-23  and  B-24  in  Appendix  B.  Table  B-23 
is  concerned  with  the  amount  of  agreement  between  graduates 
and  supervisors  as  to  which  tasks  had  been  performed, 
while  Table  B-24  is  concerned  with  the  reasons  for  non- 
performance at  the  time  of  the  field  evaluation  survey. 
Disagreements  between  supervisor  and  graduate  data  have 
often  been  cited  as  a major  reason  for  not  including  the 
graduates  themselves  in  field  evaluations  involving 
apprentice  level  resident  courses.  The  data  in  Table  B-23 
indicate  that  overall  there  was  more  agreement  than  dis- 
agreement for  79  of  the  81  job  tasks,  while  one  task  showed 
as  much  disagreement  as  agreement,  and  the  remaining  task 
showed  somewhat  more  disagreement  than  agreement.  The 
extent  of  agreement  for  the  various  tasks  ranged  from  al- 
most total  agreement  to  roughly  a fifty-fifty  split.  For 
67  of  the  job  tasks,  the  cases  of  disagreement  were  mostly 
when  the  graduates  reported  doing  the  task  but  the  super- 
visor reported  that  the  task  had  not  been  performed.  For 
ten  of  the  tasks,  the  cases  of  disagreement  were  mostly 
the  opposite,  with  supervisors  saying  the  task  was  per- 
formed and  the  graduates  reporting  that  it  was  not.  For 
four  tasks  the  directions  of  the  disagreements  were  equal. 
Several  interpretations  of  the  causes  for  the  reported  dis- 
crepancies are  possible.  It  could  be  a semantic  problem 
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on  what  does  and  does  not  constitute  performing  a task. 

More  clear-cut  descriptions  would  be  the  approach  to  this 
cause,  although  in  the  present  study,  the  majority  of  both 
the  supervisors  and  graduates  reported  having  little  diffi- 
culty with  any  of  the  instructions  or  scales.  It  could  be 
that  supervisors  are  just  not  aware  of  all  of  the  tasks 
being  performed  by  the  graduates  at  the  work  site.  Since 
over  four-fifths  of  the  tasks  were  in  the  direction  of 
more  graduates  than  supervisors  reporting  doing  the  task, 
this  seems  like  a more  plausible  explanation  of  the  data 
from  the  present  study.  The  data  on  disagreements  as  to 
whether  the  graduate  performed  a task  would  not  appear  to 
be  crucial  in  and  of  themselves.  If,  however,  they  are 
associated  with  tasks  for  which  other  indices  of  training 
adequacy  indicate  potential  problems,  then  the  reasons  for 
the  disagreements  may  throw  some  light  on  what,  if  any, 
remedial  actions  are  called  for. 

Of  more  importance  are  reported  reasons  for  non- 
performance. These  data  (Table  B - 2 4 ) were  collected  only 
from  the  supervisors  since  the  graduates  did  not  have  a 
sufficient  experience  base  upon  which  to  draw.  The  first 
column  in  Table  B-24  contains  the  number  of  graduates  not 
performing  as  reported  by  the  supervisor.  The  second 
column  indicates  the  supervisors'  opinion  that  while  the 
task  had  not  been  performed  in  the  90-120  field  evaluation 
time  frame,  it  most  likely  would  be  performed  during  the 
graduate's  first  duty  assignment  and  therefore  does  indi- 
cate an  active  training  requirement.  The  remaining  four 
columns  indicate  the  degree  of  non-requirement  for  several 
different  reasons  associated  with  the  particular  duty 
assignment  held  by  the  graduate  at  the  time  of  the  field 
evaluation . 

In  interpreting  this  type  of  data,  one  can  run  down 
the  column  indicating  the  number  not  performing  until  a 
job  task  is  reached  which  shows  an  appreciable  number. 

The  data  on  reasons  should  then  be  examined  for  such 
tasks.  If  the  bulk  of  the  reasons  are  Not  Yet  But  Will 
Be,  the  evidence  for  retaining  the  task  as  a requirement 
is  good.  To  the  extent  that  the  sum  of  the  four  Not 
Required  columns  approaches  or  exceeds  the  Not  Yet  But 
Will  Be  responses,  one  can  question  the  importance  of 
the  task  as  a training  requirement.  The  data  for  the 
course  in  the  present  study  indicate  that  for  70  of  the 
job  tasks  the  Not  Yet  But  Will  Be  responses  exceeded  the 
Not  Required  responses.  For  one  task  the  responses  were 
equally  distributed,  and  for  the  remaining  10  tasks,  the 
Not  Required  responses  exceeded  the  Not  Yet  But  Will  Be 
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responses.  All  of  these  tasks  involved  large  numbers  of 
Not  Performed  responses  so  they  should  be  examined  in  terms 
of  whether  or  not  they  constitute  a valid  training  require- 
ment. 

Since  supervisor  proficiency  ratings  play  such  an 
important  part  in  the  field  evaluation  system,  an  examina- 
tion was  made  of  the  basis  on  which  these  ratings  were 
reported  as  having  been  made.  The  distribution  of  bases, 
by  task,  is  shown  in  Table  B-25  in  Appendix  B.  These  data 
indicate  that  for  79  of  the  81  tasks,  more  supervisors 
reported  using  their  own  observations  of  task  performance 
or  review  of  task  products  as  the  basis  than  either  of  the 
other  two  bases.  For  one  of  the  tasks,  more  supervisors 
reported  using  task  information  received  from  others  than 
their  own,  and  for  one  task  the  responses  were  equally 
split.  For  no  task  did  more  than  7%  of  the  supervisors 
report  having  based  their  proficiency  ratings  only  on 
their  own  general  impressions.  An  examination  of  mean 
task  proficiency  ratings  by  major  basis  for  rating  was  made 
in  order  to  determine  the  size  and  direction  of  any  in- 
fluence that  basis  might  have  on  the  ratings  provided. 

These  data  are  shown  in  Table  B-26  in  Appendix  B.  Across 
the  81  tasks,  the  mean  ratings  based  on  own  task  obser- 
vations was  2.61  as  compared  with  a mean  rating  across 
the  81  tasks  based  on  task  information  from  others  of 
2.40.  The  mean  ratings  across  35  tasks  for  which  super- 
visors had  reported  basing  their  ratings  only  on  their 
own  general  impressions  was  2.63.  In  general,  it  appears 
that  when  supervisors  base  their  ratings  on  their  own 
observations,  whether  t as k- spec i f i c or  general,  they  tend 
to  rate  somewhat  higher  than  when  they  base  their  ratings 
on  information  they  obtain  from  others.  While  this  was 
true  on  an  overall  basis,  task-by-task  comparison  indi- 
cates that  for  17  of  the  81  tasks,  the  mean  ratings  were 
somewhat  higher  for  ratings  based  on  information  from 
others  than  on  the  supervisor's  own  observations.  While 
the  differences  in  mean  ratings  were  relatively  small  in 
the  survey  conducted  for  this  study,  it  seems  desirable 
to  hold  the  basis  for  rating  as  constant  as  possible  or 
to  identify  the  basis  and  control  for  it  in  the  inter- 
pretation of  the  data  collected. 

Measures  of  Overall  Proficiency 

Adequacy  of  training  data  based  on  percent  attain- 
ment of  at  least  the  Partially  Proficient  level  have  pre- 
viously been  presented  in  Table  B-21  and  discussed. 

Another  way  of  looking  at  overall  adequacy  is  in  terms 
of  the  calculated  average  proficiency  scores  based  on 
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supervisor  ratings.  A distribution  of  these  scores  is  pre- 
sented in  Table  17. 


TABLE  17 

Distribution  of  Calculated  Average  Proficiency  Scores 

Average  Proficiency 

Frequency 

Percent 

Cumulative 

Percent 

Below  2.00 

14 

12 

13 

2.00-2.09 

5 

5 

16 

2.10-2.19 

2 

2 

18 

2.20-2.29 

5 

4 

22 

2.30-2.39 

6 

5 

27 

2.40-2.49 

7 

6 

33 

2.50-2.59 

9 

8 

40 

2.60-2.69 

13 

11 

51 

2.70-2.79 

9 

8 

58 

2.80-2.89 

8 

7 

65 

2.90-2.99 

3 

3 

68 

3.00-3.09 

19 

16 

83 

3.10-3.19 

5 

4 

87 

3.20-3.29 

3 

3 

90 

3.30-3.39 

3 

3 

93 

3.40-3.49 

2 

2 

94 

3.50  and  above 

7 

6 

100 

N 

Mean  = 
S.D. 

Range  = 

120 

2.66 

.58 

1.29-5.00 

The  data  in  this  table  indicate  that  13%  of  the  graduates 
were  below  2.00,  the  Partially  Proficient  level;  55%  were 
between  2.00  and  2.99,  the  Partially  Proficient  level; 
and  32%  were  at  or  above  3.00,  the  Competent  level.  These 
results  may  be  compared  with  similar  results  based  on  the 
supervisors'  ratings  of  the  graduate's  overall  ability  to 
perform  duties  as  shown  in  Table  18.  In  this  case,  11% 
of  the  sample  were  in  the  Unsatisfactory  or  Marginal  cate- 
gory; 52%  were  in  the  Proficient  category;  and  37%  were 
in  the  Competent  and  Highly  Competent  categories.  The 
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TABLE  18 

Distribution  of  Overall  Ability  Ratings 


Ability  Rating 

Frequency 

Percent 

Cumulative 

Percent 

Unsatisfactory 

4 

1 

1 

1 

Marginal 

3 

12 

10 

11 

Proficient 

2 

63 

52 

63 

Competent 

1 

26 

21 

84 

Highly  competent 

0 

19 

16 

100 

N = 121 

Mean  = 1.59 

S.D.  = .90 

Range  = 0—4 


similarity  of  results  obtained  from  using  the  two  general 
criterion  measures  would  seem  to  indicate  that  either 
could  be  used  interchangeably  and  provide  evidence  of 
the  basic  reliability  of  the  proficiency  information 
collected  from  the  supervisors  in  this  study. 

Supervisor  judgments  regarding  reasons  for  reported 
1 es s - than- sat  is  factory  performance  are  shown  in  Table  19. 
Poor  attitude  is  cited  more  frequently  than  any  other 
reason,  and  inadequacy  of  resident  course  training  is 
cited  only  twice  . 


TABLE  19 

Reasons  for  Less  Than  Satisfactory  Performance 

Overall  Ability 

Reason  Reported 

Unsatisfactory 

Marginal 

No  reason  reported 

0 

1 

Inadequate  training 

0 

2 

Poor  attitude 

0 

5 

Lack  of  ability 

0 

1 

Other 

1 

3 

All  reasons 

1 

12 

( 
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Diagnostic  Follow-Up  Questionnaires 

In  an  attempt  to  obtain  from  field  supervisors  more 
detailed  information  about  job  tasks  with  which  at  least 
some  graduates  were  having  difficulty,  follow-up  evaluation 
sheets  were  sent  to  each  supervisor  who  had  rated  a 
graduate's  performance  on  one  or  more  tasks  as  less  than 
Partially  Proficient.  The  follow-up  questionnaires  were 
thus  tailor-made  from  a basic  set  of  81  job  task  follow-up 
sheets  with  each  supervisor  getting  as  many  separate  task 
sheets  as  his  1 es s - than- Part ia 1 ly - Pro f i c i ent  ratings 
warranted.  The  data  in  Table  20  indicate  that  41  super- 
visors were  sent  follow-up  questionnaires  consisting  of 
anywhere  from  1 to  42  task  sheets.  These  41  supervisors 
represented  34%  of  the  122  supervisors  who  returned  field 
evaluation  questionnaires.  The  41  follow-up  questionnaires 
contained  an  aggregate  of  419  task  sheets  which  repre- 
sented the  number  of  task  performances  in  the  total 
sample  that  had  been  rated  less  than  Partially  Proficient. 
This  is  about  4%  of  the  total  number  of  task  performances 
that  could  have  been  rated  (81  tasks  for  each  of  122 
supervisors) . Only  18  supervisors  (44%)  of  the  41  sent 
follow-up  questionnaires  returned  them  to  the  project 
staff,  and  of  these  only  15  (36%)  contained  task  specific 
responses . 


TABLE  20 

Diagnostic  Questionnaire  Administration 


Numbor  of 

Quest ionnairei  Sant  Out 

Questionnaires  Returned 

Talks  per 
Questionnaire 

Frequency 

Total  Task 
Involvement 

Frequency 

Total  Task 
Involvement 

1 

5 

5 

3 

2* 

2 

8 

16 

3 

4* 

3 

5 

15 

4 

12 

4 

3 

12 

1 

4 

5 

1 

5 

1 

5 

6 

2 

12 

0 

0 

7 

3 

21 

2 

14 

10 

1 

10 

0* 

14 

2 

28 

0 

0 

15 

2 

30 

1 

15 

16 

1 

16 

0 

0 

29 

4 

116 

2 

58 

30 

2 

60 

0 

0 

31 

1 

31 

0 

0 

42 

1 

42 

0 

0 

Total 

41 

419 

19 

114 

One  response  did  not  provide  talk  data 
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Data  regarding  the  return  rate  and  response 
follow-up  questionnaires  are  shown  in  Tables 


time  for 
21  and  22. 


TABLE  21 

Return  Rate  of  Diagnostic  Questionnaires  as  a 
of  Number  of  Tasks  per  Questionnaire 

Function 

Type  of  Response 

Diagnostic  Questionnaires  Sent  Out  With  . . . 

Total 

1-5  Tasks 

6-10  Tasks 

1 1-16  Tasks 

Over  16  Tasks 

Data  reply: 

Frequency 

10 

2 

1 

2 

15 

Percent 

45 

33 

20 

25 

37 

Non-data  reply: 

Frequency 

2 

1 

0 

0 

3 

Percent 

9 

17 

0 

0 

7 

No  reply: 

Frequency 

10 

3 

4 

6 

23 

Percent 

45 

50 

80 

75 

56 

Total. 

Frequency 

22 

6 

5 

8 

41 

Percent 

100 

100 

100 

100 

100 

TABLE  22 

Response  Time  for  Return  of  Diagnostic  Questionnaires 

Number  of  Tasks 

Number 

Number  of  Days 

per  Questionnaire 

of  Cases 

Range 

Average 

1 

3 

16-34 

22.3 

2 

3 

11-33 

22.0 

3 

4 

19-31 

23.0 

4 

1 

0 

38.0 

5 

1 

0 

19.0 

7 

2 

0 

11.0 

10 

1 

0 

31.0 

15 

1 

0 

25.0 

29 

2 

19-33 

26.0 
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The  return  rate  for  these  questionnaires  was  somewhat 
related  to  the  number  of  task  sheets  included  in  the 
questionnaire  in  the  expected  direction  of  a higher  return 
on  the  shorter  questionnaire.  But  the  return  rate  was  no 
greater  than  50%  even  for  questionnaires  having  as  few 
as  between  one  and  five  sheets.  There  was  also  a slight 
tendency  for  response  time  to  be  positively  related  to  the 
length  of  the  questionnaire.  Response  times  ranged  from 
a low  of  11  to  a high  of  38  calendar  days  against  a sug- 
gested response  date  of  four  weeks.  It  appears  that  if 
the  supervisor  is  going  to  respond,  he  will  do  so  within 
a month.  It  is  perhaps  interesting  to  note  that  while  the 
follow-up  questionnaires  were  sent  out  with  an  official 
cover  letter  on  Air  Force  stationery  over  a Colonel's 
signature,  these  questionnaires  were  the  only  ones  for 
which  the  responses  were  to  be  sent  directly  to  the  proj- 
ect staff  rather  than  to  the  Training  Evaluation  Division 
at  the  technical  training  wing.  This  may  have  had  some 
deleterious  effect  on  the  motivation  for  a supervisor  to 
complete  and  return  the  follow-up  questionnaires.  If 
the  number  of  individually  inadequate  task  performance 
ratings  are  as  low  for  most  specialties  as  they  were  in 
the  present  study,  it  appears  that  it  would  be  feasible 
to  conduct  the  follow-up  procedures  by  telephone  rather 
than  by  mail.  In  the  present  study  it  would  have  required 
only  41  telephone  calls.  Relevant  responses  to  telephone 
inquiries  would  assure  a close  to  100%  response  and  each 
could  be  probed  as  deeply  as  the  situation  seemed  to 
warrant.  Optimal  benefit  from  such  telephone  follow-ups 
could  be  obtained  only  if  the  caller  was  a person  thor- 
oughly familiar  with  both  the  training  course  and  work 
site  procedures.  Depending  upon  the  relative  qualifi- 
cations of  the  personnel  involved  at  any  particular  tech- 
nical training  wing  at  any  given  time,  this  might  mean 
using  personnel  from  the  curriculum  design  or  instruc- 
tional units  rather  than  personnel  from  the  Training  Eval- 
uation Division. 

In  reviewing  the  follow-up  questionnaire  returns,  it 
was  found  that  no  information  had  been  returned  for  23  of 
the  81  tasks.  One  follow-up  report  was  returned  for  an 
additional  29  job  tasks;  two  reports  for  16  tasks;  three 
reports  for  7 tasks;  four  reports  for  3 tasks;  five  reports 
for  1 task;  six  reports  for  1 task;  and  8 reports  for  1 
task.  It  is  quite  clear  that  this  type  of  diagnostic 
follow-up  data  is  qualitative,  not  quantitative  in  nature. 
The  follow-up  responses  were  turned  over  to  the  Training 
Evaluation  Division  for  a qualitative  analysis  by  spe- 
cialists in  the  field. 
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Supplemental  Questionnaires 

Supplemental  questionnaires  had  been  sent  to  super- 
visors in  order  to  obtain  confirmation  of  existing  train- 
ing standard  levels  and  to  generate  task  importance  indices 
that  could  be  used  in  interpreting  obtained  task  perform- 
ance data.  Table  B-27  in  Appendix  B summarizes  the  data 
from  Supplemental  Questionnaire  B having  to  do  with  the 
level  of  proficiency  designated  on  the  STS  as  required  at 
the  appropriate  level.  Notwithstanding  the  fact  that  the 
project  staff  had  received  numerous  comments  during  the 
early  field  visits  regarding  the  manner  in  which  these 
levels  were  coordinated  throughout  major  commands,  the 
supervisors  in  the  sample  for  this  study  overwhelmingly 
confirmed  the  levels  specified  in  the  STS.  For  only  one 
item  did  more  than  12%  disagree  with  the  stated  require- 
ment. For  that  one  item,  29%  disagreed  with  the  leve! 
specified,  e.g.,  lb.  Eight  wanted  the  item  upgraded  to 
2b  and  one  each  wanted  2a,  2c,  3b,  and  3c.  Field  con- 
firmation of  proficiency  levels  does  not  seem  to  be 
required  based  on  the  results  of  the  present  study. 

A direct  measure  of  task  criticality  was  requested 
on  Supplemental  Questionnaire  A.  The  results  from  this 
questionnaire  are  shown  in  Table  B-28  in  Appendix  B.  Of 
the  59  elements  involved,  14  (24%)  received  mean  criti- 
cality ratings  at  less  than  the  Important  level.  As  a 
check  on  the.  reliability  of  the  pooled  supervisor  ratings 
of  criticality,  Cronbach's  Alpha  coefficient  was  calculated 
using  the  ratings  of  34  supervisors  on  59  STS  items.  The 
calculated  Alpha  on  this  criticality  data  was  .95. 

Another  measure  of  importance  in  terms  of  the  impact 
on  the  unit  to  which  the  graduate  is  assigned  if  there 
are  performance  deficiencies  was  requested  on  Supplemental 
Questionnaire  C.  The  results  from  this  questionnaire 
are  shown  in  Table  B-29  in  Appendix  B.  For  all  tasks, 
the  extra  time  necessary  to  train  and  supervise  appren- 
tices was  listed  as  the  leading  impact  of  training  defi- 
ciencies. For  most  tasks,  delayed  skill  upgrading  was 
listed  as  the  second  most  likely  impact.  Decreased  sys- 
tem reliability  was  a relatively  close  third.  Increased 
system  downtime,  no  impact,  production  delays,  and  no 
skill  upgrading  followed  in  declining  order  of  likely 
impact.  An  average  weighted  impact  score  was  calculated 
by  using  the  weights  shown  in  Table  B-29.  These  were 
chosen  on  a judgmental  basis  by  the  project  staff.  All 
impacts  were  given  equal  weights  with  task-done-but-no- 
impact  receiving  less  weight,  and  tas k - not - requi red 
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responses  receiving  zero  weights.  Other  weighting  formulas 
may  be  derived  which  may  be  more  appropriate  for  the  imme- 
diate situation.  The  principle  of  relative  task  impact 
scores,  however,  would  remain  operative  for  all  specialties 

Two  separate  checks  on  the  reliability  of  the  pooled 
supervisor  ratings  of  the  impact  of  training  deficiencies 
were  made.  Both  checks  involved  the  ratings  from  45  super- 
visors on  59  STS  items.  Cronbach's  Alpha  was  computed 
separately  for  each  of  the  eight  separate  impact  columns. 

If  the  impact  column  was  checked,  the  item  received  a 
score  of  1 and  if  not  checked,  a score  of  0.  The  Alphas 
obtained  were  as  follows: 


Task  Not  Required  = .91 
No  Impact  = .61 
Upgrading  Impossible  = .55 
Delayed  Upgrading  = .55 
Time  of  Others  = .85 
Decreased  System  Reliability  = .69 
Increased  Downtime  = .51 
Delayed  Production  Schedules  = .53 


A split-half  procedure  was  used  to  check  the  reli- 
ability of  the  Average  Weighted  Impact  score.  One  super- 
visor was  randomly  dropped  from  the  sample  and  the 
remaining  44  supervisors  were  randomly  split  into  two 
groups.  Split-half  reliability  coefficients  were  calcu- 
lated based  on  all  of  the  separate  STS  items  and  on  the 
grouped  STS  items.  The  Pearson  coefficient  for  the  grouped 
data  was  .95  and  for-  the  separate  data  it  was  .92. 

The  interrelationships  of  the  measures  obtained  from 
the  supplemental  questionnaires  and  other  measures  are 
shown  in  Table  23.  Examination  of  this  correlation  matrix 
indicates  that  hours  of  instruction  as  specified  in  the 
relevant  Program  of  Instruction  are  significantly  cor- 
related only  with  measures  of  training  adequacy  reported 
by  graduates.  The  impact  of  training  deficiency  score  is 
significantly  correlated  with  all  variables  except  hours 
of  instruction  and  the  STS  level.  Task  criticality 
ratings  are  significantly  correlated  with  all  the  vari- 
! ables  except  hours  of  instruction.  Overall  proficiency 

ratings  are  significantly  correlated  with  all  variables  f 

except  hours  of  instruction  and  STS  level.  Tasks  not  per- 
i formed  correlates  significantly  with  all  variables  except 


hours  of  instruction.  Because  of  a scale  reversal  on 
this  variable,  the  coefficients  carrv  a minus  siun.  but 
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TABLE  23 

Intercorrelations  Between  Specialty  Training  Standard  Variables 

Hour*  of 
Instruction 

Training 

Deficiency 

Task  Cri- 
ticality 

Overall 

Proficiency 

Tasks  not 
Performed 

Training 

Adequacy 

STS 

Level 

Impact  of  training  deficiency 

12 

Task  criticality 

20 

81  •* 

Overall  proficiency 

26 

51“ 

44** 

Tasks  not  performed 

-22 

-86" 

-85" 

-53" 

Training  adequacy 

41" 

62" 

55" 

82" 

57" 

STS  level 

13 

14 

34" 

5 

-34* 

5 

Distributional  characteristics: 

N 

59 

59 

59 

42 

42 

42 

59 

Mean 

505 

1.73 

2.12 

2.62 

22 

2.00 

3.44" 

S.O. 

6.81 

.28 

46 

3C 

22.85 

.16 

1.15 

Range 

0-50 

67-1  94 

80-2.80 

189-3.09 

1-34 

1.70-2.30 

1-5 

* Statistically  significant  at  5%  level 
••  Statistically  significant  at  1%  level. 

•••  STS  scala  1b/- 1 . 1b-2.  2b/-3.  2b-4.  2c/-5. 
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the  direction  of  the  relationship  is  positive.  The  train- 
ing adequacy  measure  correlates  with  all  other  variables 
except  STS  level.  The  STS  level  scale  was  a crudely 
developed  scale  designed  to  throw  some  light  on  possible 
relationships  between  level  of  the  stated  training  require- 
ment and  the  other  variables.  The  variable  was  signifi- 
cantly related  only  to  tasks  not  performed  and  task  criti- 
cality ratings  . 

The  pattern  of  intercorrelations  of  these  variables 
leads  to  several  conclusions.  The  significant  relation- 
ships between  the  task  importance  indices,  overall  pro- 
ficiency, tasks  not  performed,  and  training  adequacy 
indicate  a strong  internally  consistent  structure  of  the 
data  base  collected  in  the  study.  Data  collected  from 
different  sources  using  a variety  of  survey  instruments 
produced  a pattern  of  indices  suggesting  that  the  instru- 
ments are,  in  fact,  measuring  only  slightly  different 
aspects  of  the  same  basic  factors  and  that  they  represent 
valid,  logical  linkages  between  the  factors. 
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Another  finding  is  that,  at  least  in  the  present 
study,  there  is  no  statistically  significant  correlation 
between  the  number  of  hours  of  instruction  allocated  in 
the  Program  of  Instruction  and  any  of  the  task  importance 
or  task  performance  indices.  This  may  raise  some  ques- 
tions about  the  allocation  of  training  hours  in  the  course 
as  presently  structured.  The  hours  of  instruction  allo- 
cations in  the  study  were  summarized  by  the  project  staff 
from  information  printed  in  the  POI . Some  assumptions 
had  to  be  made  in  handling  blocks  of  instruction  that 
covered  more  than  one  STS  item.  The  operational  rule  used 
was  to  divide  the  time  allocation  for  the  block  equally 
between  all  tasks  within  the  instructional  block.  This 
may  have  produced  some  distortion  of  the  hours  of  instruc- 
tion variable.  Also,  an  STS  item  ceases  to  be  listed  in 
the  POI  after  it  was  tested  for  criterion  level  performance 
in  the  course.  It  is  probable  that  additional  time  was 
devoted  to  the  item  after  that  point  but  was  not  included 
in  the  time  assigned  to  that  task.  Both  of  the  above  would 
lead  to  an  understatement  of  the  time,  but  since  the  rules 
were  applied  systematically  across  all  tasks  and  instruc- 
tional blocks,  this  should  not  have  dramatically  affected 
any  existing  statistical  relationships.  It  is  suggested 
that  in  future  applications,  values  o.n  the  time  allocation 
variable  be  derived  by  personnel  from  the  curriculum  design 
or  instructional  units  at  the  technical  training  wing. 

The  summary  of  time  allocations  within  the  POI  should  also 
include  those  items  which  were  not  included  in  the  field 
evaluation  because  they  involved  either  task  knowledge 
only  or  general  subject  matter  knowledge  only.  This  will 
provide  a rough  index  of  the  extent  to  which  the  course  is 
task  performance  oriented.  In  the  present  study,  approxi- 
mately 332  hours  were  allocated  to  performance  tasks  in- 
cluded in  the  field  survey,  20  hours  to  task  knowledge 
items,  and  about  332  hours  to  general  subject  matter  items. 

Detailed  Task  Evaluations 

The  present  study  attempted  to  develop  instruments 
and  procedures  that  would  provide  more  detailed  infor- 
mation for  use  in  the  feedback  step  of  the  ISD  process. 

The  data  summaries  included  in  this  section  all  involve 
the  presentation  of  specific  tasks  in  a tabular  format 
designed  to  impart  a "goodness  of  training"  or  value  in- 
dex for  each  task  in  relation  to  various  factors.  The 
concept  is  identical  for  all  of  the  tables,  the  only 
things  that  change  are  the  dimensions  represented  on  the 
horizontal  and  vertical  scales.  The  value  scoring  con- 
cept is  illustrated  by  the  illustrative  table  shown  in 
Table  24. 
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TABLE  24 

Illustrative  Decision  Table 

Dimension  One 

. 

Dimension  Twn  |_QW  ^ 

ft'  L 

" 0 

1 2 

The  assumptions  for  the  values  assigned  to  each  cell 
are  that  if  there  is,  or  it  is  felt  that  there  should  be, 
a direct  relationship  between  the  two  variables,  then  the 
diagonal  represents  a "best  match"  situation  or  an  "O.K." 
situation.  As  one  moves  away  from  the  diagonal,  the  sit- 
uation becomes  less  "0 . K . " . For  convenience  sake,  the 
cells  to  the  upper  right  of  the  diagonal  have  been  assigned 
positive  weights  and  those  to  the  lower  left  of  the  diag- 
onal have  been  assigned  negative  weights.  The  specific 
interpretation  of  the  positive  and  negative  weights  will, 
of  course,  vary  depending  upon  the  dimensions  displayed 
in  the  table. 

This  approach  may  be  used  in  a wide  variety  of  sit- 
uations to  help  identify  where  component  parts  of  a sys- 
tem fall  in  regard  to  any  set  of  relevant  dimensions.  It 
assumes  that  there  should  be  a direct  relationship  between 
the  two  dimensions  displayed  on  the  table.  The  substantive 
meaning  of  the  diagonal  in  each  table  will  depend  somewhat 
on  how  each  dimension  is  scaled.  In  the  present  study, 
the  cutting  points  for  all  dimensions  except  the  STS  level 
were  established  in  a manner  designed  to  divide  the  dis- 
tribution on  that  dimension  into  nearly  equal  quarters. 

This  is  not  critical  to  the  method  since  cutting  scores 
for  any  variable  can  be  established  on  any  logically  de- 
fensible rationale.  The  diagnostic  value  of  this  approach 
will,  however,  be  diminished  if  there  is  only  a small  dis- 
persion on  the  dimensions  displayed.  For  the  benefit  of 
the  reader,  the  scale  values  used  to  delimit  each  dimen- 
sion category  are  shown  on  the  tables. 
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Hours  of  instruction:  Tables  B-30,  B-31,  and  B-32 
in  Appendix  B display  task  numbers  in  terms  of  hours  of 
instruction  versus  three  indices  of  task  importance,  e.g., 
task  criticality,  impact  of  training  deficiencies,  and  STS 
level.  Data  for  all  of  these  variables  are  or  could  be 
made  available  without  the  time  lag  associated  with  getting 
on-the-job  performance  of  graduates.  In  many  Air  Force 
situations,  data  from  the  relevant  Occupational  Survey 
regarding  percent  performing  and  percent  of  time  spent  per- 
forming could  be  used  as  other  or  substitute  task  import- 
ance indices.  In  order  to  do  this,  however,  it  would  be 
necessary  to  equate  the  job  tasks  on  the  Occupational  In- 
ventory and  the  items  on  the  relevant  Specialty  Training 
Standard.  This  equating  task  has  already  been  done  for  some 
but  not  all  specialties.  The  data  in  these  tables  can  be 
interpreted  in  terms  of  adequate,  over-,  and  under- training, 
keeping  in  mind  that  these  are  only  gross  indices.  Never- 
theless, one  can  examine  in  further  detail  those  tasks 
which  are  farthest  out  of  line  to  see  if  there  are  justi- 
fiable reasons  for  their  being  where  they  are.  Inter- 
pretations may  be  tentative  and  indicative,  certainly  not 
definitive  solely  on  the  basis  of  the  tabular  data.  The 
tables  are  designed  to  help  experienced  personnel  make  eval- 
uative judgments,  not  to  make  those  judgments  for  them. 

Tables  B-33,  B-34,  and  B-35  in  Appendix  B display 
hours  of  instruction  against  dimensions  collected  in  the 
field  survey.  These  tables  could  not  be  constructed  un- 
til after  graduates  had  been  out  in  the  field.  Again, 
data  from  a current  relevant  Occupational  Survey  regarding 
percent  of  incumbents  performing  could  be  used  in  place 
of  the  tasks  not  performed  field  data  used  in  the  present 
study.  This  again  would  require  a matching  of  Occupational 
Inventory  job  tasks  and  STS  items. 

Overall  proficiency:  Tables  B-36,  B-37,  B-38,  and 
B-39  in  Appendix  B display  rated  overall  proficiency 
against  the  other  measures.  Since  rated  proficiency  is 
dependent  upon  job  performance,  these  analyses  could  not 
be  made  until  sometime  after  completion  of  the  resident 
course . 

Task  criticality  versus  impact  of  training  deficiency: 
The  data  in  Table  B-40  in  Appendix  B show’s  the  relation- 
ship between  the  two  task  importance  indices  generated 
from  field  data.  This  is  merely  a different  way  for  dis- 
playing the  relationship  shown  in  terms  of  statistical 
indices  in  the  correlation  matrix  presented  in  Table  23. 
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Detailed  interpretations  of  specific  data  included 
in  these  simple  decision  table  formats  of  the  data  collected 
in  this  study  have  not  been  presented  because  they  would 
be  specific  to  the  specialty  and  the  resident  course  in- 
volved. Only  persons  thoroughly  familiar  with  the  specialty 
could  make  definitive  recommendations  from  such  data. 

The  data  are,  however,  designed  to  be  a useful  input  to  such 
personnel.  A helpful  step  is  to  display  all  of  the  indi- 
ces associated  with  a given  task  on  a summary  sheet  such 
as  that  shown  in  Table  B-41  in  Appendix  B. 
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DEPARTMENT  OF  THE  AIR  FORCE 
HEADQUARTERS  CHANUTE  TECHNICAL  TRAINING  CENTER  (ATC) 
CHANUTE  AIR  FORCE  BASE.  ILLINOIS'  6IS68 


REPLY  TO 
ATTN  OF 


3350  TCHTW/TTS 


*u»jtcT  Field  Evaluation  Questionnaire  - Crs  No.  C3ABR42330  000,  Aircraft  Electrical 
Systems  Specialist 


TO: 


1.  Pursuant  to  AFR  50-38,  Field  Evaluation  of  Formal  School  Graduates, 
this  Headquarters  is  conducting  a field  evaluation  of  the  training 
received  by  selected  graduates  of  subject  course. 

2.  It  is  requested  that  the  IMMEDIATE  SUPERVISOR  OF 

complete  the  attached  question- 
naire and  answer  sheets  (ATC  Forms  1612A  and  B). 

3.  If  the  supervisor  cannot  comply  with  this  request,  please  return  the 
questionnaire  and  answer  sheets,  and  annotate  this  letter  to  show  the 
reason  for  noncompletion.  If  the  graduate  has  been  reassigned,  please 
provide  the  address  of  the  new  unit  of  assignment. 

4.  The  Project  Officer,  Mr.  Carter,  may  be  contacted  via  AUT0V0N  at 
862-2119  for  further  guidance  or  additional  information  regarding  the 
questionnaire. 

5.  Use  the  Inclosed  self-addressed  envelope  to  return  the  questionnaire 

and  answer  sheets  through  the  Military  Postal  System  before  

Also,  place  your  organizational  address  and  office  symbol  in  the  upper 
left-hand  corner  of  the  return  envelope. 

FOR  THE  COMMANDER 


Commander,  3350  Technical  Training  Wing 


1 Atch 

Questionnaire  & Answer 
Sheets  w/Envelope 
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Record  No. 


80806 
Survey  ID. 


DEPARTMENT  OF  THE  AIR  FORCE 
HQ  3350TH  TECHNICAL  TRAINING  WING  (ATC) 
CHANUTE  AIR  FORCE  BASE,  ILLINOIS  61868 


FIELD  EVALUATION  QUESTIONNAIRE 
FOR 

COURSE  NO.  C3ABR42330  000,  AIRCRAFT  ELECTRICAL 
SYSTEMS  SPECIALIST 


NOTICE  TO  SUPERVISOR 

1.  Your  assistance  Is  requested  in  rating  the  job  performance  of  this  airman 
who  is  under  your  supervision.  Two  separate  answer  sheets  (ATC  Forms  I6I2A  and 
B)  are  provided  for  responses  to  the  first  part  of  the  questionnaire.  Responses 
to  the  second  part  of  the  questionnaire  dealing  with  graduate  proficiency  should 
be  made  directly  on  the  questionnaire  Itself.  This  format  Is  being  field  tested 
as  part  of  a research  and  development  project  being  conducted  by  the  American 
Institutes  for  Research  under  a contract  for  the  Air  Force  Human  Resources  Labor- 
atories. Please  return  both  the  survey  form  and  ATC  Forms  I6I2A  and  B to  the 
above  address.  Space  Is  provided  at  the  back  of  the  questionnaire  for  comments 
you  wish  to  add. 

2.  In  the  space  provided  on  ATC  Form  I6I2A,  please  print  your  name,  duty  phone, 
and  AUT0V0N  number. 

3.  Please  use  only  a No.  2 pencil  for  marking  ATC  Forms  I6I2A  and  B. 

4.  IT  IS  IMPORTANT  THAT  ATC  FORMS  A AND  B BE  HANDLED  CAREFULLY.  Staple  holes, 
frayed  corners,  paper  clip  impressions,  extraneous  pencil  marks,  creases,  and 
light  pencil  responses  will  cause  the  scanning  machine  to  reject  the  answer  sheets. 


This  survey  Is  being  conducted  under 
the  authority  of  AFR  50-38  and  is 
excluded  from  the  approval  provisions 
of  AFRs  30-23  and  178-7. 
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Answer  ONLY  the  following  Items  on  ATC  Form  I6I2A. 

I-II  - Blacken  the  space  corresponding  to  Graduate's  Current  Grade. 

1-12  - Blacken  the  spaces  that  Identify  Graduate's  Duty  AFSC.  First  column 
represents  prefix  (if  any),  the  next  five  represent  AFSC  number,  and 
the  last  column  represents  suffix  (if  any). 

1-19  - Identify  Graduate's  Primary  AFSC.  Instructions  are  the  same  as  for 
1-12  above. 

1-26  - Identify  the  Graduate's  Duty  Assignment  by  blackening  one  space  that 
corresponds  to  one  of  the  following  numbers  on  the  answer  sheet: 

(0)  Apprentice  Aircraft  Electrical  Systems  Specialist. 

(1)  Aircraft  Electrical  System  Specialist. 

12)  Aircraft  Electrical  Systems  Technician. 

(3)  Other  (indicate  appropriate  title);  


1-27  - Identify  which  of  the  following  most  closely  describes  the  tasks  per- 
formed by  the  Graduate  in  your  unit  since  graduation  by  blackening  one 
space  that  corresponds  to  one  of  the  following  numbers  on  ATC  I6I2A: 

(0)  Spent  most  time,  either  by  himself  or  with  more  experienced  personnel, 
performing  battery  maintainence  functions. 

(1)  Spent  most  time,  either  by  himself  or  with  more  experienced  personnel, 
visually  inspecting  and  troubleshooting  electrical  circuits  and  systems 

on  aircraft.  Spent  relatively  little  time  performing  shop  related  tasks. 

(2)  Spent  most  time,  either  by  himself  or  with  more  experienced  personnel, 
perform! nq  general  shop  and  bench  checking  tasks,  such  as  assembly, 
disassembly,  and  Internal  repair  tasks. 

(3)  Spent  most  time,  either  by  himself  or  with  more  experienced  personnel, 
performing  troubleshooting  and  general  shop  functions. 

1-29  - Blacken  this  space  If  the  Graduate  is  enrolled  in  or  has  completed 
CDC  42350,  Aircraft  Electrical  Systems  Specialist  (AFSC  42350). 
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1-30  - Blacken  this  space  if  the  Graduate  is  attending  or  has  completed  a 

Field  Training  Detachment  (FTD)  course(s),  and  enter  course(s)  identifi- 
cation here: 


1-31  - Blacken  the  space  that  properly  Identifies  this  Graduate, 
thru 

1-32  - 1-31  - Female 
1-32  - Male 

1-63  - Are  the  female  personnel  assigned  to  your  shop  physical ly  capable  of  per- 
forming all  of  the  tasks  in  this  AFS  that  the  male  personnel  normally 
perform?  Blacken  the  space  for  either  YES  or  NO. 

If  you  marked  NO,  please  list  those  tasks  which  the  female  personnel  are 
not  capable  of  performing:  


Answer  ONLY  the  following  items  on  ATC  Form  I6I2B. 

2-15  - Did  the  Graduate  have  any  significant  OJT  difficulties?  If  you  marked 
YES,  please  explain  difficulties.  


2-16  - Rate  Graduate's  overall  ability  to  perform  assigned  duties  by  blackening 
the  appropriate  space. 

2-17  - If  you  rated  the  Graduate  MARGINAL  or  UNSATISFACTORY  In  Item  2-16, 

blacken  the  one  reason  on  the  answer  sheet  which  accounts  for  the  low 
rating.  If  OTHER  is  marked,  please  list  reason(s)  here: 


2-18  - Do  you  have  suggestions  for  Improving  the  course?  If  you  marked  YES, 
please  list  your  comments  on  the  last  page  of  the  questionnaire. 

2-19  - Blacken  the  space  corresponding  to  your  Current  Grade. 
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2-20  - Blacken  the  spaces  that  Identify  your  Duty  AFSC.  First  column  represents 
prefix  (If  any),  the  next  five  represent  AFSC  number,  and  the  last 
column  represents  suffix  (if  any). 

2-27  - Blacken  the  appropriate  spaces  to  Identify  your  Primary  AFSC.  Instructions 
are  the  same  as  for  2-20  above. 

2-35  - Blacken  a number  in  each  column  to  indicate  the  months  you  have  been  a 

supervisor  in  the  423X0  specialty.  For  example:  6 months  =06.  If  more 
than  99,  blacken  both  9’s  and  enter  total  months  here: 


2-40  - Blacken  the  space  corresponding  to  your  Major  Command  or  Agency. 

2-41  - Blacken  the  one  space  that  indicates  the  number  of  female  423X0  personnel 
assigned  to  you r shop. 

2-45  - Blacken  the  spaces  (numbers)  that  identify  your  SSAN. 

2-54  - Blacken  spaces  to  indicate  the  date  you  completed  this  questionnaire. 


(S) 


GRADUATE  JOB  PROFICIENCY 


In  this  part  of  the  questionnaire,  you  are  to  mark  your  answers  directly  on  the 
questionnaire  itself.  In  order  to  follow  up  on  any  questions  that  may  arise  on 
this  experimental  part  of  the  questionnaire,  please  Indicate  your  mailing 
address  below: 

Name  and  Rank:  SSAN:  

Mailing  Address:  


Zip  Code: 


The  Job  tasks  listed  In  this  part  of  the  questionnaire  were  selected  from 
Specialty  Training  Standard  (STS)  423X0,  dated  July  1976.  Following  each  task 
statement  Is  the  STS  paragraph  number.  We  are  Interested  In  getting  from  you 
three  types  of  information  about  each  of  the  tasks  listed. 


First,  did  or  did  not  the  graduate  perform  the  task  listed  since  his  assignment 
to  his  present  duty  position.  Since  the  resident  course  trains  for  the  overall 
AFSC  and  not  for  a specific  duty  assignment,  and  since  It  has  been  only  a short 
time  since  his  completion  of  the  course.  It  Is  highly  unlikely  that  all  graduates 
will  perform  all  of  the  tasks  listed.  One  purpose  Is  to  find  out  how  many  and 
which  tasks  graduates  are  likely  to  be  assigned  during  the  first  90-120  days 
after  graduation.  If  the  graduate  performed  the  task,  put  a check  mark  In 
Column  A,  If  not,  put  a check  mark  In  Column  B.  If  you  checked  Column  B then 
put  a check  mark  or  marks  in  Columns  C,  D,  E,  F,  G which  indicate  one  or  more 
reasons  why  he  hasn't  performed  It.  Definitions  for  these  column  headings  are: 


Column  C 


Not  relevant  to  Job.  This  task  Is  not  part  of  the  Job  at  the 
work  sita  to  which  the  graduate  has  been  assigned. 


Column  D - Not  yet,  but  will  be.  The  graduate  has  not  yet  been  asked  to  do 
this  task,  but  more  than  likely  he  will  be  required  to  do  It 
before  completion  of  his  first  duty  assignment. 

Column  E - Not  a 1 1 owed ♦ The  graduate  Is  not  assigned  to  do  this  task 

because  of  supervisory  practices  at  this  work  site.  He  might  be 
assigned  this  task  If  assigned  to  a different  work  site. 

Column  F - Not  requl red-hardware.  This  task  Is  relevant  only  to  hardware 
or  weapon  systems  which  are  not  present  at  this  work  site.  They 
might  be  at  different  work  sites. 
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Column  G - Not  requ i red-other . If  there  Is  any  other  reason  why, 

check  here  and  write  In  the  reason  in  the  comoents  section 
at  the  end  of  the  questionnaire. 

If  you  checked  Column  B,  and  one  or  more  of  Column's  C-G,  do  not  make  any  more 
ratings  for  that  task.  Proceed  to  the  next  task. 


If  you  checked  Column  A (Yes),  you  are  to  indicate  your  judgment  of  the  graduate' 
proficiency.  Before  you  do  that,  however,  please  indicate  the  primary  basis  for 
your  judgment.  Do  this  by  putting  a check  in  either  Column  H,  I , or  J.  The 
definitions  for  these  Columns  are: 

Column  H - Own  task  observation.  Use  this  if  you  have  personally  seen 

the  graduate  perform  the  task  or  a highly  sirr’lar  task,  or  if 
you  have  personally  seen  the  product  of  h vork  on  this  task 
or  a highly  similar  task. 

Column  I - Task  information  from  others.  Use  this  if  - t 1 ive  not  person- 
ally seen  the  graduate  perform,  but  are  basing  /our  rating  on 
information  you  got  from  other  supervisors  or  experienced  Journey 
men  who  have  seen  the  graduate  perform  this  task  or  a highly 
s iml lar  task. 


Column  J - General  impression.  Use  this  if  you  have  not  observed  the 
graduate  actual ly  perform  this  or  a highly  similar  task  and 
are  basing  your  judgment  on  a general  opinion,  based  on  obser- 
vation of  overall  activities  at  this  work  site. 


Indicate  your  judgment  of  the  graduate's  proficiency  level  by  putting  a check  in 
one  of  the  Columns  K-0.  The  definitions  for  these  Columns  are: 


Column  K - I ncapab le.  Can  do  no  part  of  the  task  without  being  told 

or  shown  how  to  do  It.  This  rating  applies  only  to  assigned 
tasks. 

Column  L - Extremely  Limited.  Can  do  simple  parts  of  the  task.  Needs 
to  be  told  or  shown  how  to  do  most  of  the  task. 


Column  M - Partially  Proficient.  ’Can  do  most  parts  of  the  task.  Needs 
to  be  helped  on  hardest  parts. 

Column  N - Competent.  Can  do  a I I parts  of  the  task.  Needs  only  a spot 
check  of  completed  work. 

Column  0 - Highly  Proficient.  Can  do  the  complete  task  quickly  and 
accurately.  Can  tell  others  how  to  do  the  task. 
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Performed  or  Not  Performed  Basil  for  I Proficiency  Level 

I Reason  Why  Not  Rating  I Demonstrated 
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pr  Not  Parfocnwd  I Basil  (or  I Proficiency  Level 

Reason  Why  Noi  R*«i"9  Oamonstralad 


I 

1 

> 


(S) 


REACTIONS  TO  SURVEY  FORMAT 

Since  some  of  the  procedures  and  formats  Included  In  the  Questionnaire,  you  have 
just  completed  are  experimental  and  are  being  field  tested  by  the  American 
Institutes  for  Research  under  contract  to  the  Air  Force  Human  Resources  Laboratory, 
we  are  interested  in  your  reactions  to  the  questionnaire. 

I.  Did  the  Instructions  and  questionnaire  format  give  you  a clear  Idea  of  what 
you  were  to  do? 

Yes No 

If  no,  what  needs  to  be  clarified? 


2.  Were  the  descriptions  of  the  rating  scales  and  column  headings  clear  to  you? 

Yes No 

If  no,  which  descriptions  need  to  be  revised? 


3.  Which  of  the  following  best  describes  your  preference  for  recording  your 
answers  on  separate  answer  sheets  or  on  the  questionnaire  Itself? 

A.  I very  much  prefer  to  use  separate  answer  sheets. 

B.  I somewhat  prefer  to  use  separate  answer  sheets. 

C.  It  makes  no  difference  to  me. 

D.  I somewhat  prefer  to  write  my  answers  on  the  questionnaire  Itself. 

E.  I very  much  prefer  to  write  my  answers  on  the  questionnaire  Itself. 

4.  About  how  long  did  It  take  you  to  complete  this  questionnaire? 

minutes. 


5.  Have  you  ever  been  required  In  the  past  to  fill  out  a field  evaluation 

questionnaire  as  the  supervisor  of  a recent  resident  course  graduate?  <g- 

No Yes  If  yes,  how  many?  

V 


PLEASE  USE  THE  REVEPSE  SIDE  OF  THIS  PAGE  TO  MAKE  ANY  OTHER  COMMENTS  YOU  MAY  FEEL 
ARE  APPROPRIATE  ABOUT  THE  USE  OF  THIS  TYPE  OF  A FIELD  QUESTIONNAIRE  TO  GATHER 
DATA  REGARDING  HOW  WELL  THE  BASIC  RESIDENT  COURSE  HAS  PREPARED  AIRMEN  TO  PERFORM 
THE  TASKS  LIKELY  TO  BE  ASSIGNED  TO  THEM  ON  THEIR  FIRST  DUTY  ASSIGNMENT. 
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Record  No. 


80805 
Survey  ID. 


DEPARTMENT  OF  THE  AIR  FORCE 
HQ  3350TH  TECHNICAL  TRAINING  WING  (ATC) 
CHANUTE  AIR  FORCE  BASE,  ILLINOIS  61868 


FIELD  EVALUATION  QUESTIONNAIRE 
FOR 

COURSE  NO.  C3ABR42330  000,  AIRCRAFT  ELECTRICAL 
SYSTEMS  SPECIALIST 


NOTICE  TO  GRADUATE 


1.  Your  assistance  Is  requested  In  providing  -Information  regarding  the  extent  to 
which  you  have  used  Knowledges  and  skills  acquired  in  Course  3ABR  42330  since 
completion  of  the  course.  ATC  Form  161 IA  provided  herein.  Is  for  your  responses 
to  the  first  part  of  the  questionnaire.  The  format  for  the  second  part  of  the 
questionnaire  Is  being  field  tested  as  part  of  a research  and  development  project 
by  the  American  Institutes  for  Research  under  contract  to  the  AF  Human  Resources 
Laboratory.  You  are  to  record  your  responses  for  the  second  part  of  the  question- 
naire directly  on  this  questionnaire  Itself.  Space  Is  provided  at  the  end  of  this 
questionnaire  for  comments  you  may  wish  to  add. 

2.  In  the  space  provided  on  ATC  Form  161 IA,  please  print  your  name,  duty  phone, 
and  AUT0V0N  number. 

3.  Please  use  only  a No.  2 pencil  for  marking  ATC  Form  161 IA. 

4.  IT  IS  IMPORTANT  THAT  ATC  FORM  161 IA  BE  HANDLED  CAREFULLY.  Staple  holes, 
frayed  corners,  paper  clip  impressions,  extraneous  pencil  marks,  creases,  and 
light  pencil  responses  will  cause  the  scanning  machine  to  reject  the  answer  sheets. 


This  survey  Is  being  conducted  under 
the  authority  of  AFR  50-38  and  Is 
excluded  from  the  approval  provisions 
of  AFR's  30-23  and  178-7. 


(G) 


Answer  ONLY  the  following  Items  on  ATC  Form  161  I A 


l-ll  - Blacken  the  space  corresponding  to  your  current  grade. 

1-14  - Blacken  the  spaces  that  Identify  your  Duty  AFSC.  First  column  represents 
prefix  (if  any),  the  next  five  represent  AFSC  number,  and  the  last  column 
represents  suffix  (if  any). 

If  you  received  any  type  of  formal  training  since  you  completed  course  3ABR42330, 
write  In  the  course  title  and  length,  and  blacken  the  corresponding  number  on  ATC 
Form  161  I A.  If  you  have  not  received  any  formal  training  since  completing  the 
course,  black  in  space  1-46. 


1-37 


-38 

Course  Title 

Length 

in 

Weeks 

-39 

Course  Title 

Length 

I n 

Weeks 

-40 

Course  Title 

Length 

In 

Weeks 

-41 

Course  Title 

Length 

I n 

Weeks 

Course  Title 

Length 

1 n 

Weeks 

1-46  - No  formal  training  since  completion  of  course  3ABR42330. 

1-47  - Have  you  experienced  any  significant  difficulty  In  performing  your  duties 
or  on-the-job  training  that  you  consider  the  result  of  Inadequate  training 
In  this  course?  If  yes,  explain  in  the  Comments  Section  at  the  end  of  this 
questionnaire. 

1-48  - Blacken  In  the  space  which  corresponds  to  the  Major  Command  or  Agency  to 
which  you  are  assigned. 

Blacken  In  one  of  the  following  spaces  which  most  closely  describes  the  types  of 

tasks  you  have  been  doing  since  assignment  to  a unit  after  completion  of  course 

3ABR42330. 

1-49  - Spent  most  time,  either  by  yourself  or  with  more  experienced  personnel, 
performing  battery  maintenance  functions. 

1-50  - Spent  most  time,  either  by  yourself  or  with  more  experienced  personnel, 
visually  Inspecting  and  troubleshooting  electrical  circuits  and  systems 
on  aircraft.  Spent  relatively  little  time  performing  shop  related  tasks. 
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1-51  - Spent  most  time,  either  by  yourself  or  with  more  experienced  personnel, 
performing  general  shop  and  bench  checking  tasks,  such  as  assembly, 
disassembly,  and  internal  repair  tasks. 

1-52  - Spent  most  time,  either  by  yourself  or  with  more  experienced  personnel, 
performing  troubleshooting  and  general  shop  functions. 


(G) 


ADEQUACY  OF  TRAINING  ON  INDIVIDUAL  TASKS 

The  following  pages  contain  significant  job  tasks  at  the  3 skill  level  selected 
from  Specialty  Training  Standard  (STS)  423X0,  July  1976.  For  each  task  listed 
you  are  to  indicate  whether  or  not  you  performed  the  task,  either  as  part  of  your 
proficiency  OJT  activities  or  as  part  of  your  normal  duty  assignment,  whether  or 
not  the  task  was  involved  in  the  Career  Development  Course  (CDC)  for  upgrading  to 
the  5 skill  level,  or  whether  or  not  it  was  Involved  in  any  Field  Training  Detach- 
ment (FTD)  Course  you  may  have  taken  since  completion  of  course  3ABR42330. 

If  you  have  not  performed  the  task,  and  it  was  not  involved  in  any  CDC  or  FTD 
training,  you  are  to  check  the  "No"  column  under  headings  "Performed  Task",  "Used 
In  CDC  Work",  and  "Used  In  FTD”.  However,  if  you  performed  the  task,  or  If  It  was 
involved  in  your  CDC  work,  or  it  was  involved  in  FTD  activities  in  which  you 
participated,  you  are  to  Indicate  your  opinion  about  how  well  the  resident  course 
prepared  you.  Use  the  following  scale  for  indicating  your  opinion. 

1 = The  resident  course  did  not  provide  me  with  the  level  of  knowledge  and 

skill  I needed  to  perform  assigned  job  tasks  at  the  level  expected  of 
me,  or  to  complete  the  CDC  in  the  expected  time  period,  or  to  satisfac- 
torily complete  FTD  courses  in  which  I participated. 

2 = The  resident  course  provided  me  with  the  level  of  knowledge  and  skill 

I needed  to  perform  assigned  job  tasks  at  the  level  expected  of  me, 
or  to  complete  the  CDC  In  the  expected  time  period,  or  to  satisfactorily 
complete  FTD  courses  In  which  I participated. 

3 = The  resident  course  provided  me  with  a higher  level  of  knowledge  and 

skill  than  I needed  to  perform  assigned  job  tasks  at  the  level  expected 
of  me,  or  to  complete  the  CDC  in  the  expected  time  period,  or  to  satis- 
factorily complete  FTD  courses  In  which  I participated. 


(G) 

RATING  YOUR  TRAINING  ON  INDIVIDUAL  TASKS 


TASK 

Performed 

task 

Used  in 
CDC  work 

Used  in 

FTD 

No 

If  yes. 
adequacy 

No 

If  yes. 
adequacy 

No 

If  yes. 
adequacy 

1 

2 

3 

1 

2 

3 

1 

2 

3 

SHOP  AND  FLIGHT  LINE  SAFETY 

1.  Apply  safety  precautions  when  using  tools  and  equip- 
ment. (3a) 

2.  Practice  housekeeping  consistent  with  safety  of  per- 
sonnel and  equipment.  (3b) 

3.  Apply  safety  precautions  pertaining  to  aircraft 
electrical  system.  (3d (5)) 

4.  Apply  FOD  Instructions.  (3f) 

TECHNICAL  PUBLICATIONS 

5.  Locate  technical  order  numbers  and  titles  In  Index 
type  technical  orders.  (4a) 

6.  Use  technical  manuals  as  a source  of  Information  for 
performing  maintenance  and  Inspections.  (4b) 

7.  Use  abbreviated  technical  orders  when  performing 
Inspections  and  maintenance.  (4d) 

8.  Locate  maintenance,  management,  and  administrative 
information  In  methods  and  procedures  technical 
orders.  (4e) 

MAINTENANCE  AND  INSPECTION  SYSTEM  AND  FORMS 

9.  Use  maintenance  data  collection  forms.  (7d) 

ELECTRICAL  MAINTENANCE  FUNDAMENTALS 

10.  Use  handtools.  (9a) 

II.  Care  for  handtools.  (9a) 

12.  Solder  electrical  connections  to  Include  solid-state 
devices  and  printed  circuits.  (9b) 

13.  Use  solder  less  electrical  connector  devices.  (9c) 

14.  Perform  wire  maintenance.  (9d) 

15.  Use  safety Inq  devices.  (9q) 

' 

16.  Select  hardware.  (9h) 

17.  Use  hardware.  (9h) 

METERS  AND  TESTERS 

18.  Use  frequency  meters.  (I3a(l)) 

19.  Care  for  frequency  meters.  (I3a(l)) 

. 

20.  Use  multimeters.  (I3a(2)) 

21.  Care  for  multimeters.  (I3a(2)) 

22.  Use  vacuum-tube  voltmeters.  ( 1 3a < 3 ) ) 

23.  Care  for  vacuum-tube  voltmeters.  (I3a(3)) 

24.  Use  oscl 1 loscope.  ( 1 3a < 5 ) ) 

25.  Care  for  oscilloscope.  ( 1 3a ( 5 ) ) 

TEST  STANDS  AND  LOADBANKS 

26.  Inspect  generator  test  stands.  ( 1 4a ( 2 ) ) 

27.  Operate  generator  test  stands.  ( I 4a ( 2 ) ) 

28.  Inspect  inverter  test  stands.  ( 1 4c ( 2 ) ) 

29.  Operate  inverter  test  stands.  ( 1 4c ( 2 ) ) 

30.  1 Aspect  loadbanks.  (I4d(2)) 

31.  Operate  loadbanks.  ( 1 4d (2) ) 

_ 
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(G) 


RATING  YOUR  TRAINING  ON  INDIVIDUAL  TASKS 


TASK 

Performed 

Task 

Used  in 
CDC  work 

Used  in 

FTD 

No 

If  yet. 
adequacy 

No 

If  vet. 
adequacy 

No 

If  yes. 
adequacy 

1 

2 

3 

1 

2 

3 

1 

2 

3 

AIRCRAFT  BATTERY  SHOP  OPERATION 

32.  Service  lead  acid  batteries.  ( 1 5a ( 1 ) ) 

33.  Maintain  lead  acid  batteries.  (I5a(l)) 

34.  Service  alkaline  batteries.  (I5a(2)) 

35.  Maintain  alkaline  batteries.  (I5a(2)) 

36.  Use  charging  equipment.  ( 1 5b ( 2) ) 

37.  Maintain  charging  equipment.  (15b (3)) 

38.  Use  servicing  equipment.  (I5c(l)) 

~ 

39.  Maintain  servicing  equipment.  (I5c(2)) 

AIRCRAFT  POWER  SYSTEMS 

40.  Operate  DC  generator  system.  (I6a(2)) 

41.  Inspect  OC  generator  system.  (16a (2)) 

42.  Troubleshoot  DC  generator  system.  (16a (2)) 

43.  Operate  AC  generator  system.  ( 1 6b (2 ) ) 

44.  Inspect  AC  generator  system.  (16b (2)) 

45.  Troubleshoot  AC  generator  system.  (16b (2)) 

46.  Operate  battery  system.  (I6c(2)) 

47.  Inspect  battery  system.  (16c (2)) 

48.  Troubleshoot  battery  system.  (J6c(2)) 

49.  Operate  transformer-rectifier  system.  (I6d(2)) 

50.  Inspect  transformer-recti f ler  system.  (I6d(2)) 

51.  Troubleshoot  transformer-rectifier  system.  (I6d(2)) 

52.  Operate  rotary  Inverter  system.  ( 1 6e ( 2 ) ) 

53.  Inspect  rotary.  Inverter  system.  (I6e(2)) 

54.  Troubleshoot  rotary  inverter  system.  (!6e(2)) 

AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS 

55.  Operate  lighting  system.  (17a (2)) 

56.  Inspect  lighting  system.  (I*7a(2)) 

57.  Troubleshoot  lighting  system.  ( 1 7a (23 > 

58.  Operate  landing  gear  system.  (17b (2)) 

59.  Inspect  landing  gear  system.  ( 1 7b (2 ) ) 

60.  Troubleshoot  landinq  qear  system.  ( 1 7b ( 2 ) ) 

61.  Operate  trim  system.  (I7e(2)(a)) 

62.  Inspect  trim  system.  ( 1 7e (2) (a ) ) 

63.  Troubleshoot  trim  system.  (I7e(2)(a)) 

64.  Operate  flap  control  and  Indicating  system. ( l7e(2T(b) 

65.  Inspect  flap  control  and  Indicating  system. ( I7e(2) (b) 

66.  Troubleshoot  flap  control  and  Indicating  system. 

( I7e(2) (b) ) 

~ 

67.  Operate  fire  warning  system.  (I7f(2)(a)) 

68.  Inspect  fire  warninq  system.  (I7f(2)(a)) 

69.  Troubleshoot  fire  warninq  system.  (I7f(2)(a)) 

70.  Operate  overheat  warning  system.  ( 1 7f (2) (b ) ) 

71.  Inspect  overheat  warninq  system.  (I7f(2)(b)i 

72.  Troubleshoot  overheat  warninq  system.  (I7f(2)(b)> 

73.  Operate  takeoff  warning  system.  ( 1 7f (2) (c) ) 

74.  Inspect  takeoff  warning  system.  (I7f(2)(c)) 

I 
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RATING  YOUR  TRAINING  ON  INDIVIDUAL  TASKS 


Performed 

Task 


Used  in 
COC  work 


Used  in 
FTO 


I If  yes. 

No  | adequacy 


AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS  (continued) 

75.  Troubleshoot  takeoff  warninq  system.  (I7f(2)(c)) 


76.  Operate  master  warninq  system.  (I7f(2) (d)> 


77.  Inspect  master  warninq  system.  (I7f(2)(d)) 


78.  Troubleshoot  master  warninq  system.  (I7f(2)(d)) 


79.  Operate  iet  enqlne  startlnq  and  Iqnitlon.  (I7h(2)(b 


80.  Inspect  Jet  enqlne  startlnq  and  Iqnition.  (I7h(2)(b) 


81.  Troubleshoot  jet  engine  starting  and  ignition. 
( I7h(2) (b) ) 


If  you  have  suggestions  or  recommendations  for  improving  training  In  this  course, 
list  your  comments  here  and  reference  them  to  the  appropriate  numbered  entries 
in  the  questionnaire. 


(G) 


REACTIONS  TO  SURVEY  FORMAT 

Since  some  of  the  procedures  and  formats  Included  In  the  questionnaire  you  have 
just  completed  are  experimental  and  are  being  field  tested  by  the  American 
Institutes  for  Research  under  contract  to  the  Air  Force  Human  Resources  Laboratory, 
we  are  Interested  in  your  reactions  to  the  questionnaire. 

I.  Did  the  instructions  and  questionnaire  format  give  you  a clear  Idea  of  what 
you  were  to  do? 

Yes No 

If  no,  what  needs  to  be  clarified? 


2.  Were  the  descriptions  of  the  rating  scales  and  column  headings  clear  to  you? 

Yes No 

If  no,  which  descriptions  need  to  be  revised? 


3.  Which  of  the  following  best  describes  your  preference  for  recording  your 
answers  on  separate  answer  sheets  or  on  the  questionnaire  Itself? 

A.  I very  much  prefer  to  use  separate  answer  sheets. 

B.  I somewhat  prefer  to  use  separate  answer  sheets. 

C.  It  makes  no  difference  to  me. 

D.  1 somewhat  prefer  to  write  my  answers  on  the  questionnaire  Itself. 

E.  I very  much  prefer  to  write  my  answers  on  the  questionnaire  Itself. 

4.  About  how  long  did  It  take  you  to  complete  this  questionnaire? 

minutes. 

PLEASE  USE  THE  REVERSE  SIDE  OF  THIS  PAGE  TO  MAKE  ANY  OTHER  COMMENTS  YOU  MAY  FEEL 
ARE  APPROPRIATE  ABOUT  THE  USE  OF  THIS  TYPE  OF  A FIELD  QUESTIONNAIRE  TO  GATHER 
DATA  REGARDING  HOW  WELL  THE  BASIC  RESIDENT  COURSE  HAS  PREPARED  AIRMEN  TO  PERFORM 
THE  TASKS  LIKELY  TO  BE  ASSIGNED  TO  THEM  ON  THEIR  FIRST  DUTY  ASSIGNMENT. 
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FIELD  EVALUATION  QUESTIONNAIRE 

FOR  ' 

COURSE  NO.  C3ABR42330  000,  AIRCRAFT  ELECTRICAL 
SYSTEMS  SPECIALIST 

j 

I.  As  part  of  a field  test  of  a possible  revised  field  evaluation  questionnaire  \ 

format,  opinions  of  supervisors  are  being  sought  regarding  the  relative  criticality 

of  tasks  listed  in  the  Specialty  Training  Standard  (STS)  for  the  3 skill  level.  I 

All  are  required  at  one  or  another  work  sites  to  which  resident  course  graduates 

in  this  AFSC  may  be  assigned,  otherwise  they  wouldn’t  be  set  forth  as  training  J 

standards.  Nevertheless,  some  are  undoubtedly  more  important  than  others  In  any 

given  work  site  in  that  the  Impact  on  unit  mission  accomplishment  Is  greater 

if  there  Is  a performance  deficiency  on  some  of  the  tasks  than  others.  \ 


2.  Your  assistance  Is  requested  In  indicating  your  opinion  of  the  relative 
criticality  of  the  tasks  listed.  Make  you r judgments  of  criticality  In  terms 
of  the  particular  duty  assignment  held  by  the  recent  graduate  of  the  resident 
course  whose  proficiency  you  just  rated  since  It  Is  illustrative  of  the  kind 
of  assignment  held  by  3 skill  level  apprentices  during  their  first  duty  assignment. 


3.  Record  your  judgments  on  the  questionnaire  itself  using  the  following  scale. 

0 ~ Task  is  not  particularly  relevant  to  the  airman’s  assignment  In 

that  it  is  rarely  If  ever  performed  by  an  incumbent  at  this  level. 

1 = Task  Is  of  limited  relevance  to  the  airman’s  assignment.  It  Is 

nice- to- know  and  useful  but  not  essential  in  that  an  airman  can 
satisfactorily  perform  his  duty  assignment  without  being  able  to 
perform  it  satisfactorily. 


► 


V 


2  = Task  is  important  to  the  airman's  assignment  In  that  ability  to 
perform  it  satisfactorily  will  enhance  mission  accomplishment  of 
the  unit  to  which  he  Is  assigned. 


3 


Task  is  critical  to  the  airman's  assignment  In  that  mission  accomplishment 
of  the  unit  to  which  he  is  assigned  cannot  be  attained  If  he  cannot 
perform  It  satisfactorily. 
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(A) 


TASK  CRITICALITY 


TASK 

LEVEL  OF 
CRITICALITY 

A 

is 

0 

E | 

1 

ii 

II 

2 

u Task  is 
critical 

3.  SHOP  AND  FLIGHT  LINE  SAFETY 

a.  Apply  safety  precautions  when  using  tools  and  equipment 

b.  Practice  housekeeping  consistent  with  safety  of  personnel  and 
equipment 

d.  Apply  safety  precautions  pertaining  to 
(1)  Enqine  air  intake  and  exhaust 

(2)  High  intensity  sound 

(3)  Propeller  and  rotor  planes  of  rotation  n 

(4)  High  voltage  and  antenna  radiation 

(5)  Aircraft  electrical  system 

(6)  Power  actuated  surfaces  and  equipment 

e.  Use  portable  fire  extinguishers 

f.  Apply  FOD  Instructions 

4.  TECHNICAL  PUBLICATIONS 

a.  Locate  technical  order  numbers  and  titles  in  index  type 
technical  orders 

b.  Use  technical  manuals  as  a source  of  information  for 
performlnq  maintenance  and  inspections 

c.  Apply  instructions  in  Time  Compliance  Technical  Orders 

d.  Use  abbreviated  technical  orders  when  performing  Inspections 
and  maintenance 

e.  Locate  maintenance,  management  and  administrative  Information 

In  methods  and  procedures  technical  orders 

7.  MAINTENANCE  AND  INSPECTION  SYSTEM  AND  FORMS 
d.  Use  maintenance  data  collection  forms 

9.  ELECTRICAL  MAINTENANCE  FUNDAMENTALS 
a.  Use  and  care  for  handtools 

b.  Solder  electrical  connections  to  include  solid-state 
devices  and  printed  circuits 

c.  Use  solder  less  electrical  connector  devices 

d.  Perform  wire  maintenance 

q.  Use  safety inq  devices 

h.  Select  and  use  hardware 

13.  METERS  AND  TESTERS 
a.  Use  and  care  for: 

( 1 ) Frequency  meters 

(2)  Multimeters 

(3)  Vacuum-tube  voltmeters 

(4)  Tube  tester  n 

(5)  Oscilloscope 

(8)  Frequency  counters 

(9)  Wheatstone  bridge 

14.  TEST  STANDS  AND  LOAD  BANKS 
a.  Generator  test  stands 
(2)  Inspect  and  operate 
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TASK  CRITICALITY 


(A) 


TASK 

LEVEL  OF 
CRITICALITY 

Not 

relevant 

if 

e! 

jS 

C 

-5 
-*  Q 

28  t 
2 

u Task  is 
critical 

14.  TEST  STANDS  AND  LOAD  BANKS  (continued) 
b.  AC  control  panel  test  sets 
(2)  Inspect  and  operate 

c.  Inverter  test  stands 
(2)  Inspect  and  operate 

d.  Load  banks 

(2)  Inspect  and  operate 

15.  AIRCRAFT  BATTERY  SHOP  OPERATION 
a.  Service  and  maintain 
(1)  Lead  acid  batteries 

(2)  Alkaline  batteries 

b.  Charging  equipment 
(2)  Use 

( 3 ) Mai nta i n 

c.  Servicing  equipment 
(I)  Use 

(2)  Maintain 

16.  AIRCRAFT  POWER  SYSTEMS 
a.  DC  generator  system 

(2)  Operate,  inspect  and  troubleshoot 

b.  AC  generator  system 

(2)  Operate,  Inspect  and  troubleshoot 

c.  Battery  system 

(2)  Operate,  Inspect  and  troubleshoot 

d.  Transformer-recti f ier  system 

(2)  Operate,  inspect  and  troubleshoot 

e.  Inverter  system 

(2)  Operate,  inspect  and  troubleshoot 
(a)  Rotary 

17.  AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS 
a.  Lighting 

(2)  Operate,  Inspect  and  troubleshoot 

b.  Landing  gear 

(2)  Operate,  Inspect  and  troubleshoot 

c.  Antiskid 

(2)  Operate,  Inspect  and  troubleshoot 

d.  Nosewheel  steering 

(2)  Operate,  inspect  and  troubleshoot 

e.  Secondary  flight  controls 

(2)  Operate,  Inspect  and  troubleshoot 
(a)  Trim 

(b)  Flap  control  and  Indicating 

TASK  CRITICALITY 


LEVEL  OF 
CRITICALITY 


,8  I 

lilt  ll 

3S  k.I  1=  5 

1 2 3 


17.  AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS  (cont 

f.  Warning 

(2)  Operate,  inspect  and  troubleshoot 

(a)  Fire 


(b)  Overheat 


(c)  Takeoff 


(d)  Master  warn  in 


(e)  Fuel 


g.  Fuel  control 

(2)  Operate,  inspect  and  troubleshoot 


h.  Power  plant  electrical 

(2)  Operate,  Inspect  and  troubleshoot 
(a)  Control  circuits 


(b)  Jet  engine  starting  and  Ignition 


i.  Nesa  Glass  anti-icing 

(2)  Operate,  Inspect,  and  troubleshoot 


(continued) 


REACTIONS  TO  SURVEY  FORMAT 


Since  the  format  of  the  questionnaire  you  have  just  completed  Is  experimental 
and  is  being  field  tested  by  the  American  Institutes  for  Research  under  contract 
to  the  Air  Force  Human  Resources  Laboratory,  we  are  interested  in  your  reactions 
to  the  questionnaire. 

I.  Did  the  instructions  and  questionnaire  format  give  you  a clear  Idea  of 
what  you  were  to  do? 

Yes  No  

If  no,  what  needs  to  be  clarified? 


2.  Were  the  descriptions  of  the  rating  scales  and  column  headings  clear  to  you? 

Yes  No  

If  no,  which  descriptions  need  to  be  revised? 


3.  Which  of  the  following  best  describes  your  feelings  about  the  task  you  just 
completed? 

A.  I felt  comfortable  doing  it  and  believe  I was  able  to  make  generally 

reliable  and  valid  judgments. 

B.  I felt  a little  uneasy  since  I'm  not  sure  a person  in  my  position  can 

make  the  type  of  judgments  required,  but  I believe  I was  able  to  make 
reasonably  reliable  and  valid  Judgments. 

C.  I felt  very  uneasy  since  I don't  believe  a person  in  my  position  can 

make  the  type  of  judgments  required  reliably  and  validly,  but  I tried 
to  supply  the  best  data  I could. 

D.  I felt  extremely  uneasy  since  I feel  rather  strongly  that  a person  In 

my  position  cannot  make  the  type  of  judgments  required,  so  my  data 
should  probably  not  be  given  much  weight. 

4.  About  how  long  did  It  take  you  to  complete  this  questionnaire? 

minutes. 


PLEASE  USE  THE  REVERSE  SIDE  OF  THIS  PAGE  TO  MAKE  ANY  OTHER  COMMENTS  YOU  MAY 

FEEL  ARE  APPROPRIATE  ABOUT  THE  USE  OF  THIS  TYPE  OF  A QUESTIONNAIRE  WITH  FIRST-LINE 

SUPERVISORS. 


Record  No. 


80806  (B) 
Survey  ID. 
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FIELD  EVALUATION  QUESTIONNAIRE 
FOR 

COURSE  NO.  C3ABR42330  000,  AIRCRAFT  ELECTRICAL 
SYSTEMS  SPECIALIST 

!•  As  part  of  a field  test  of  a possible  revised  field  evaluation  questionnaire 
format,  opinions  of  supervisors  are  being  sought  regarding  the  validity  of  the 
levels  of  knowledge/proficiency  specified  for  the  3 skill  level  in  the  Specialty 
Training  Standard  (STS),  July  1976.  For  your  convenience,  the  Proficiency  Code 
Key  used  on  the  STS  is  shown  below. 


PROFICIENCY  CODE  KEY 

SCALE 

VALUE 

DEFINITION:  Tha  lnd..iduol 

Ml 

u 

3- 

OL  J 

m 

A. 

i 

Con  do  simple  ports  of  the  task.  Needs  to  be  told  or  shown  Sow  to  do  most  of  the  task. 

'(EXTREMELY  LIMITED) 

2 

Con  do  most  parts  of  the  task.  Needs  help  only  on  hardest  per?*.  May  not  meet  local  demands  for 
speed  or  accurocy.  (PARTIALLY  PROFICIENT) 

3 

Can  do  all  parts  of  the  task.  Needs  only  a seo*  check  of  comoie'ed  work.  Meets  minimum  local 
demands  for  speed  end  occwracy.  (COMPETENT) 

A 

Can  do  the  complete  tosk  quickly  end  accurately.  Con  roil  or  show  others  how  to  do  the  task. 

(HIGHLY  PROrlCIENT' 

m 

8- 
H uj  -J 

rss 

M 

o 

Can  name  parts,  tools,  ond  simple  facts  about  the  task.  (NCVcNCLATURE) 

I 

Con  dotormine  stea  by  step  procedures  for  do  mg  the  tesk.  PROCEDURES' 

c 

(OPERATING  PRINCIPLES) 

4 

Can  predict,  identify,  ond  rotolva  problems  about  tha  task.  (COMPLETE  THEORY) 

SIS 
• * 

A 

Can  identify  basic  tacit  and  t«f*s  about  tha  subject.  (FACTS) 

B 

Con  esploin  relationship  of  basic  facts  ond  stafa  genorol  principles  about  tha  subjocf.(PRlNClPLES) 

C 

Con  onolyta  facts  ond  principles  ond  draw  conclusions  about  rha  subject.  (ANALYSIS) 

D 

Can  evaluate  conditions  ond  moko  proper  decisions  about  the  subject.  (EVALUATION) 

- EXPLANATIONS  - 

S • A tosh  knowledge  scole  volve  may  bo  used  alone  or  with  o tesk  porTormonce  scole  volue  to  define  • level  of 
knowledge  foe  o specific  to«k.  (Eiomplet:  b o nd  lb) 

•#  A subject  knowledge  scale  valve  is  used  alone  to  define  o level  of  knowledge  for  e subject  not  directly  related  to 
any  specific  task,  or  for  o subject  common  to  several  tasks. 

• This  mark  is  used  alone  instead  of  0 scale  value  to  show  that  no  proficiency  trominp  is  provided  in  the  course, 
or  that  no  proficiency  is  required  at  this  skill  level. 

X This  mark  »S  used  olene  »n  course  columns  to  show  that  troinmg  is  not  given  due  to  lim*toi>ons  in  resources. 

(B) 


2.  The  knowledge/proficiency  level  specified  by  the  STS  for  the  3 skill  level 
for  selected  tasks  from  the  STS  is  shown  on  the  following  pages.  Please 
indicate  whether  or  not  you  feel  the  level  specified  is  a realistic  expectation 
for  3 skill  level  apprentices  approximately  90  to  120  days  after  completion  of 
the  basic  resident  course.  Record  your  Judgments  on  the  questionnaire  In  terms 
of  the  particular  duty  assignment  held  by  the  recent  graduate  whose  proficiency 
you  just  rated. 

3.  If  you  feel  the  knowledge/proficiency  level  presently  specified  Is  about 
right,  put  a check  mark  in  the  "O.K."  column.  If  not,  put  a check  mark  In  the 
"Not  O.K."  column  and  then  write  In  the  level  you  feel  would  be  more  realistic 
in  the  "Should  Be"  column.  Keep  In  mind  current  manpower  and  budgetary 
constraints  when  making  your  judgments. 


TRAINING  LEVEL  STANDARDS 


TASK  / KNOWLEDGE 

Prevent 

STS 

Laval 

OK. 

Not 

OK 

Shouk 

B« 

3.  SHOP  AND  FLIGHT  LINE  SAFETY 

a.  Apply  safety  precautions  when  usinq  tools  and  equipment 

2b 

b.  Practice  housekeeping  consistent  with  safety  of  personnel 
and  equipment 

2b 

d.  Apply  safety  precautions  pertaining  to 
(1)  Enqine  air  intake  and  exhaust 

2c/c 

(2)  Hlqh  intensity  sound 

2c/c 

(3)  Propeller  and  rotor  planes  of  rotation 

2c/c 

(4)  High  voltaqe  and  antenna  radiation 

2c/c 

(5)  Aircraft  electrical  system 

2b 

(6)  Power  actuated  surfaces  and  equipment 

2c/c 

e.  Use  portable  fire  extinquishers 

"2b7b 

f.  Apply  FOD  instructions 

2b 

4.  TECHNICAL  PUBLICATIONS 

a.  Locate  technical  order  numbers  and  titles  in  index  type 
technical  orders 

2b 

b.  Use  technical  manuals  as  a source  of  information  for 
performlnq  maintenance  and  inspections 

2b 

c.  Apply  instructions  in  Time  Compliance  Technical  Orders 

"2b7b 

d.  Use  abbreviated  technical  orders  when  performing  Inspec- 
tions and  maintenance 

2b 

e.  Locate  maintenance,  management  and  administrative  infor- 
mation in  methods  and  procedures  technical  orders 

2b 

7.  MAINTENANCE  AND  INSPECTION  SYSTEM  AND  FORMS 
d.  Use  maintenance  data  collection  forms 

2b 

9.  ELECTRICAL  MAINTENANCE  FUNDAMENTALS 
a.  Use  and  care  for  handtools 

2b 

b.  Solder  electrical  connections  to  include  solid-state 
devices  and  printed  circuits 

2b 

c.  Use  solder  less  electrical  connector  devices 

2b 

d.  Perform  wire  maintenance 

'2b 

q.  Use  safetyinq  devices 

lb 

h.  Select  and  use  hardware 

2b 

13.  METERS  AND  TESTERS 
a.  Use  and  care  for: 

( 1 ) Frequency  meters 

2b/ la 

(2)  Multimexers 

2b 

(3)  Vacuum-tube  voltmeters 

2b 

(4)  Tube  tester 

2b 

(5)  Osci 1 loscope 

'ib 

(8)  Frequency  counters 

7b7F 

(9)  Wheatstone  bridqe 

2b7T 

14.  TEST  STANDS  AND  LOAD  BANKS 
a.  Generator  test  stands 
(2)  Inspect  and  operate 

2b 

b.  AC  control  panel  test  sets 
(2)  Inspect  and  operate 

2b 

(B) 


TRAINING  LEVEL  STANDARDS 


TASK / KNOWLEDGE 

PrwMnt 

STS 

L*u»l 

O.K. 

Not 

O.K. 

Should 

B* 

14.  TEST  STANDS  AND  LOAD  BANKS  (continued) 
c.  Inverter  test  stands 
(2)  Inspect  and  operate 

lb 

d.  Load  banks 

(2)  Inspect  and  operate 

lb 

15.  AIRCRAFT  BATTERY  SHOP  OPERATION 
a.  Service  and  maintain 
(1)  Lead  acid  batteries 

2b 

(2)  Alkaline  batteries 

2b 

b.  Charging  equipment 
(2)  Use 

lb 

(3)  Maintain 

Tb7tT 

c.  Servicing  equipment 
(1)  Use 

lb 

(2)  Maintain 

lb 

16.  AIRCRAFT  POWER  SYSTEMS 
a.  DC  generator  system 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

b.  AC  generator  system 

(2)  Operate,  inspect  and  troubleshoot 

2b 

c.  Battery  system 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

d.  Transformer-recti f ler  system 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

e.  Inverter  system 

(2)  Operate,  Inspect  and  troubleshoot 
(a)  Rotary 

lb 

17.  AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS 
a.  Lighting 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

b.  Landing  gear 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

c.  Antiskid 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

d.  Nosewheel  steering 

(2)  Operate,  inspect  and  troubleshoot 

2b 

e.  Secondary  flight  controls 

(2)  Operate,  Inspect  and  troubleshoot 
(a)  Trim 

2b 

(b)  Flap  control  and  Indicating 

2b 

f . Warn  1 ng 

(2)  Operate,  Inspect  and  troubleshoot 
(a)  Fire 

2b 

(b)  Overheat 

2b 

(c)  Takeoff 

2b 

(d)  Master  warning 

2b 

(e)  Fuel 

T575T 

' 
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REACTIONS  TO  SURVEY  FORMAT 

Since  the  format  of  the  questionnaire  you  have  Just  completed  Is. experl  mental 
and  Is  being  field  tested  by  the  American  Institutes  for  Research  under  contract 
to  the  Air  Force  Human  Resources  Laboratory,  we  are  Interested  In  your  reactions 
to  the  questionnaire. 

I.  Did  the  Instructions  and  questionnaire  format  give  you  a clear  Idea  of 
what  you  were  to  do? 

Yes  No 

If  no,  what  needs  to  be  clarified? 


2.  Were  the  descriptions  of  the  rating  scales  and  column  headings  clear  to  you? 

Yes  No 

If  no,  which  descriptions  need  to  be  revised? 


3.  Which  of  the  following  best  describes  your  feelings  about  the  task  you  Just 

completed? 

A.  1 felt  comfortable  doing  It  and  believe  I was  able  to  make  generally 

reliable  and  valid  Judgments. 

B.  I felt  a little  uneasy  since  I’m  not  sure  a person  In  my  position  can 

make  the  type  of  Judgments  required,  but  1 believe  I was  able  to  make 
reasonably  reliable  and  valid  Judgments. 

C.  I felt  very  uneasy  since  I don’t  believe  a person  In  my  position  can 

make  the  type  of  Judgments  required  reliably  and  validly,  but  1 tried 
to  supply  the  best  data  I could. 

D.  I felt  extremely  uneasy  since  I feel  rather  strongly  that  a person  In 

my  position  cannot  make  the  type  of  judgments  required,  so  my  data 
should  probably  not  be  given  much  weight. 

4.  About  how  long  did  It  take  you  to  complete  this  questionnaire? 

minutes. 

PLEASE  USE  THE  REVERSE  SIDE  OF  THIS  PAGE  TO  MAKE  ANY  OTHER  COMMENTS  YOU  MAY  FEEL 

ARE  APPROPRIATE  ABOUT  THE  USE  OF  THIS  TYPE  OF  A QUESTIONNAIRE  WITH  FIRST-LINE 

SUPERVISORS. 
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DEPARTMENT  OF  THE  AIR  FORCE 
HQ  3350TH  TECHNICAL  TRAINING  WING  (ATC) 
CHANUTE  AIR  FORCE  BASE,  ILLINOIS  61868 


Supplement  to 

FIELD  EVALUATION  QUESTIONNAIRE 
FOR 

COURSE  NO.  C3ABR42330  OOO,  AIRCRAFT  ELECTRICAL 
SYSTEMS  SPECIALIST 

I.  As  part  of  a field  test  of  a possible  revised  field  evaluation  questionnaire 
format,  opinions  of  supervisors  are  being  sought  regarding  the  Impact  on  their 
unit  If  3 skill  level  apprentices  who  are  recent  graduates  of  the  basic  resident 
course  do  not  demonstrate  the  knowledge/proficiency  level  for  tasks  specified  In 
the  Specialty  Training  Standard  (STS).  To  refresh  your  memory,  the  Proficiency 
Code  Key  used  In  the  STS  Is  shown  below. 


PROFICIENCY  CODE  KEY 

SCALC 

valuc 

DEFINITION:  Th.  Individual 

01 

u 

5 

K l * 

< o > 
k 2 w 
or  -J 
«« 

ft. 

i 

Can  da  limpla  part.  al  tna  task.  Naadt  la  ba  laid  or  »hown  how  la  da  matt  ai  tha  fa*k. 

(EXTREMELY  LIMITED) 

2 

Con  do  most  ports  of  the  task  Needs  help  only  on  hardest  ports.  May  not  moot  locel  dr~:i:s  for 
speed  or  accuracy.  (PARTIALLY  PROFICIENT) 

3 

Con  do  oil  ports  of  the  task.  Needs  only  o spot  check  of  completed  work.  Meets  mimmu-  ccol 
demands  for  speed  and  accuracy.  (COMPETENT) 

i 

'HIGHLY  PROFICIENT' 

Ml 

8 - 
3 ii 

►*  g ■» 

• I J 
u 

0 

Con  nemo  ports,  tools,  and  simple  facts  about  the  task.  (NOMENCLATURE) 

b 

Con  determine  tteo  by  step  procedures  far  doing  the  totk.  (PROCEDURES 

c 

(OPERATING  PRINCIPLES) 

4 

Con  predict,  identify,  end  resolve  problems  about  the  task.  (COMPLETE  THECRV 

Z u J 

• M 

A 

Con  identify  basic  facts  ond  terms  about  the  subject.  (FACTS) 

B 

Con  eaplain  relationship  of  basic  facts  and  state  yenerel  principles  about  the  suhect.  N'C  FLES. 

C 

Con  analyte  focts  end  principles  ond  drew  conclusions  about  the  subject.  (ANALYSIS' 

D 

Con  evaluate  conditions  end  moke  proper  decisions  about  the  subject.  (EVALUATION' 

- EXPLANATIONS  - 

• A talk  knowledge  ic«U  volue  mo y bo  used  olono  or  with  m totk  perormonce  seal#  volue  to  dtfiAt  o level  of 
knowledge  (or  m specific  took.  (Eaemples:  b • nd  lb) 

••  A subject  knowledge  tcolo  value  it  utod  oloao  to  domino  o lovol  ml  knowltdfo  lot  0 subject  not  directly  rels'ed  to 

Ony  specific  task,  or  lor  o subject  common  to  tovotol  totkt. 

- Tbit  e*erk  it  used  olono  instood  of  0 tcolo  voluo  to  show  thot  no  proficiency  troininf  it  provided  m the  course, 

|;  or  that  no  proficiency  is  required  of  this  skill  lovol. 

| X This  mark  it  used  olono  in  course  columns  to  show  *hoi  troming  it  not  given  due  to  limitations  m resources 
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(C) 


2.  The  know  I edge/ prof  lei ency  level  specified  by  the  July  1976  STS  for  the  3 
skill  level  Is  shown  on  the  attached  list  of  tasks.  For  each  task  listed, 
check  the  probable  impact  on  your  uni r If  graduates  approximately  90  to  120  days 
after  completion  of  the  resident  course  do  not  demonstrate  the  level  specified. 
Record  your  judgments  on  the  questionnaire  in  terms  of  the  particular  duty 
assignment  held  by  the  recent  graduate  whose  proficiency  you  Just  rated. 

3.  Check  as  many  impact  areas  as  you  feel  are  appropriate.  The  definitions 
of  the  Impact  Deficiency  columns  are  as  follows: 

Not  required.  No  Impact  since  the  task  is  not  required  by  the  Incumbent 
of  this  particular  duty  assignment. 

No  Impact.  The  task  is  required  but  performance  deficiencies  of 
apprentices  would  have  no  appreciable  Impact  on  the  unit. 

Upgrading  Impossible.  Without  the  level  specified,  the  incumbent 
could  not  handle  required  CDC  and  OJT  assignments  In  order  to 
upgrade  to  the  5 skill  level. 

Upgrading  delayed.  The  incumbent  could  upgrade  to  the  5 skill  level 
but  would  take  longer  than  usual. 

Time  of  others.  Time  of  present  5,  7,  and  9 level  personnel  normally 
available  for  operational  duties  would  be  diverted  in  order  to 
provide  additional  training  or  supervision  of  the  apprentices. 

Reduced  reliability.  Reliability  of  unit  hardware  and/or  weapon 
systems  would  be  reduced. 

Increased  down  time.  Out-of-service  time  of  unit  hardware  and/or 
weapon  systems  would  be  increased. 

Delayed  schedules.  Unit  production  schedules  would  likely  be 
de I ayed . 


(C) 


TASK/KNOWLEDGE 

Present 

STS 

Level 

Impact  of  Deficiency  of 

3 Skill  Level  Apprentices 

Not  required 

No  impact 

Upgrading 

impossible 

Upgrading 

delayed 

° .. 
II 

£ o 

Reduced 

reliability 

Increased 

downtime 

Delayed 

schedule 

3.  SHOP  AND  FLIGHT  LINE  SAFETY 

a.  Apply  safety  precautions  when  using  tools  and 
equipment 

2b 

b.  Practice  housekeeping  consistent  with  safety  of 
personnel  and  equipment 

2b 

d.  Apply  safety  precautions  pertaining  to 
(1)  Enqine  air  intake  and  exhaust 

2c /c 

(2)  Hiqh  intensity  sound 

2c/c 

(3)  Propeller  and  rotor  planes  of  rotation 

2c/c 

(4)  Hiqh  voltage  and  antenna  radiation 

2c/c 

(5)  Aircraft  electrical  system 

2b 

(6)  Power  actuated  surfaces  and  equipment 

2c/c 

e.  Use  portable  fire  extinquishers 

2b/b 

f.  Apply  FOD  instructions 

2b 

4.  TECHNICAL  PUBLICATIONS 

a.  Locate  technical  order  numbers  and  titles  In 
index  type  technical  orders 

2b 

b.  Use  technical  manuals  as  a source  of  information 
for  performing  maintenance  and  inspections 

2b 

c.  Apply  instructions  in  Tima  Compliance  Technical 
Orders 

2b/b 

d.  Use  abbreviated  technical  orders  when  performing 
inspections  and  maintenance 

2b 

e.  Locate  maintenance,  management  and  administrative 
information  in  methods  and  procedures  technical 
orders 

2b 

7.  MAINTENANCE  AND  INSPECTION  SYSTEM  AND  FORMS 
d.  Use  maintenance  data  collection  forms 

2b 

9.  ELECTRICAL  MAINTENANCE  FUNDAMENTALS 
a.  Use  and  care  for  handtools 

2b 

b.  Solder  electrical  connections  to  include  solid- 
state  devices  and  Drinted  circuits 

2b 

c.  Use  solder  less  electrical  connector  devices 

2b 

d.  Perform  wire  maintenance 

2b 

q.  Use  safety inq  devices 

lb 

h.  Select  and  use  hardware 

2b 

13.  METERS  AND  TESTERS 
a.  Use  and  care  for: 

( 1 ) Frequency  meters 

2b/ la 

(2)  Multimeters 

2b 

73)  Vacuum-tube  voltmeters 

2b 

(4)  Tube  tester 

2b 

( 5 ) Osc i 1 1 oscope 

lb 

(8)  Frequency  counters 

1 b/b 

(C) 


TASK /KNOWLEDGE 

Present 

STS 

Level 

Impact  of  Deficiency  of 

3 Skill  Level  Apprentices 

Not  required 

No  impact 

Upgrading 

impossible 

Upgrading 

delayed 

O o 

II 

O 

if 

1,1 
cr  £ 

Increased 

downtime 

si 

i? 

O X 

13.  METERS  AND  TESTERS  (continued) 
a.  Use  and  care  for: 

(9)  Wheatstone  bridge 

2b/b 

14.  TEST  STANDS  AND  LOAD  BANKS 
a.  Generator  test  stands 
(2)  Inspect  and  operate 

2b 

b.  AC  control  panel  test  sets 
(2)  Inspect  and  operate 

2b 

c.  Inverter  test  stands 
(2)  Inspect  and  operate 

lb 

d.  Load  banks 

(2)  Inspect  and  operate 

lb 

15.  AIRCRAFT  BATTERY  SHOP  OPERATION 
a.  Service  and  maintain 
(1)  Lead  acid  batteries 

2b 

(2)  Alkaline  batteries 

2b 

b.  Charging  equipment 
(2)  Use 

lb 

(3)  Maintain 

Ib/b 

c.  Servicing  equipment 
(1)  Use  ' 

lb 

( 2 ) Mai nta i n 

lb 

16.  AIRCRAFT  POWER  SYSTEMS 
a.  DC  generator  system 

(2)  Operate,  inspect  and  troubleshoot 

2b 

b.  AC  generator  system 

(2)  Operate,  Inspect  and  troubleshoot 

2b 

c.  Battery  system 

(2)  Operate,  inspect  and  troubleshoot 

2b 

d.  Transformer-recti f ier  system 

(2)  Operate,  inspect  and  troubleshoot 

2b 

e.  Inverter  system 

(2)  Operate,  inspect  and  troubleshoot 
a.  Rotary 

lb 

17.  AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS 
a.  Lighting 

(2)  Operate,  inspect  and  troubleshoot 

2b 

b.  Landing  gear 

(2)  Operate,  inspect  and  troubleshoot 

2b 

c.  Antiskid 

(2)  Operate,  inspect  and  troubleshoot 

2b 

d.  Nosewheel  steering 

(2)  Operate,  inspect  and  troubleshoot 

2b 
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Impact  of  Deficiency  of 
3 Skill  Level  Apprentices 


TASK/KNOWLEOGE 


1 ill?. 

? I ml  Zz  II  if 

o o s|  si  il  si 

Z Z O.E  3-q  l-o  IS 


AIRCRAFT  CONTROL  AND  WARNING  SYSTEMS  (continued) 
e.  Secondary  flight  controls 

(2)  Operate,  inspect  and  troubleshoot 


control  and  I nd I cat  in 


(a)  Trim 


(b)  Fla 


Warning 

(2)  Operate,  inspect  and  troubleshoot 
(a)  Fire 


(b)  Overheat 


(c)  Takeoff 


(d)  Master  warn  in 


(e)  Fuel 


Fuel  control 

(2)  Operate,  Inspect  and  troubleshoot 


Power  plant  electrical 
(2)  Operate,  inspect  and  troubleshoot 
(a)  Control  circuits 


(b)  Jet  engine  starting  and  ignition 


Nesa  glass  anti-icing 

(2)  Operate,  inspect,  and  troubleshoot 


Delayed 


(C) 


REACTIONS  TO  SURVEY  FORMAT 

Since  the  format  of  the  questionnaire  you  have  Just  completed  Is  experimental 
and  is  being  field  tested  by  the  American  Institutes  for  Research  under  contract 
to  the  Air  Force  Human  Resources  Laboratory,  we  are  interested  in  your  reactions 
to  the  questionnaire. 

I.  Did  the  instructions  and  questionnaire  format  give  you  a clear  idea  of 
what  you  were  to  do? 

Yes No 

If  no,  what  needs  to  be  clarified? 


2.  Were  the  descriptions  of  the  rating  scales  and  column  headings  clear  to  you? 

Yes  No 

If  no,  which  descriptions  need  to  be  revised? 


3.  Which  of  the  following  best  describes  your  feelings  about  the  task  you  just 
completed? 

A.  I felt  comfortable  doing  It  and  believe  I was  able  to  make  generally 

reliable  and  valid  judgments. 

B.  I felt  a little  uneasy  since  I'm  not  sure  a person  In  my  position  can 

make  the  type  of  judgments  required,  but  I believe  I was  able  to  make 
reasonably  reliable  and  valid  judgments. 

C.  I felt  very  uneasy  since  I don't  believe  a person  in  my  position  can 

make  the  type  of  judgments  required  reliably  and  validly,  but  I tried 
to  supply  the  best  data  I could. 

D.  I felt  extremely  uneasy  since  I feel  rather  strongly  that  a person  in 

my  position  cannot  make  the  type  of  judgments  required,  so  my  data 
should  probably  not  be  given  much  weight. 

4.  About  how  long  did  It  take  you  to  complete  this  questionnaire? 

minutes. 

PLEASE  USE  THE  REVERSE  SIDE  OF  THIS  PAGE  TO  MAKE  ANY  OTHER  COMMENTS  YOU  MAY  FEEL 
ARE  APPROPRIATE  ABOUT  THE  USE  0r  '"-MS  TYPE  OF  A QUESTIONNAIRE  WITH  FIRST-LINE 
SUPERVISORS. 
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REPLY  TO 
ATTN  OF  TTS 


DEPARTMENT  OF  THE  AIR  FORCE 
HEADQUARTERS  CHANUTE  TECHNICAL  TRAINING  CENTER  < ATC ) 
CHANUTE  AIR  FORCE  BASE.  ILLINOIS  61868 


subject  Follow-Up  on  Field  Evaluation  Questionnaire  for  Course  No.  C3ABR42330  000, 
Aircraft  Electrical  Systems  Specialist 


TO 


1.  Thank  you  for  completing  and  returning  the  Field  Evaluation  Question- 
naire on  subject  course.  Your  responses  will  be  combined  with  those  of 
supervisors  of  other  recent  graduates  of  the  course.  (This  survey  Is  being 
conducted  under  the  authority  of  AFR  50-38  and  is  excluded  from  the  approval 
provisions  of  AFRs  30-23  and  178-7.) 

2.  As  part  of  a field  test  of  a modified  field  evaluation  system,  we  are 

following  up  on  those  tasks  on  which  you  rated  Airman  , 

Record  No.  , Incapable  of  performing  at  the  required  level.  A 

separate  sheet  Is  Inclosed  for  each  task  on  which  you  rated  the  graduate's 
performance  as  Inadequate.  Please  answer  the  questions  and  return  the 
separate  task  sheet (s).  Feel  free  to  use  the  back  of  the  task  sheet (s)  to 
Include  any  comments  and  opinions  regarding  task  performance  and  training 
that  you  feel  may  help  the  Technical  School  to  do  a more  effective  Job  In 
supplying  operational  units  with  graduates  who  can  perform  at  the  skill 
levels  reflected  in  the  Specialty  Training  Standard. 


3.  You  may  contact  Mr.  Carter  via  AUTOVON  at  862-2119  for  further  guidance 
or  additional  Information  regarding  the  task  sheet (s). 


4.  Use  the  Inclosed  self-addressed  envelope  to  return  the  task  sheet (s) 
through  the  Military  Postal  System  before  . 


FOR  THE  COMMANDER 


0 

A*-  E " ‘ 

/ I Depvi 


/]  J.  EDWARD  LORENZ,  Colonel,  USAF 
Deputy  Commander  for  Technical 


ning 


1 Atch 

Task  Sheet (s) 
w/Envelope 


no 


TABLE  B-1 

Overall  Ability  to  Perform  versus  General  Aptitude 

General  Aptitude 

Score 

Missing 

Unsatis- 

factory 

Marginal 

Satisfactory 

Very  satis- 
factory 

Excellent 

91-100 

5 

0 

1 

2 

3 

i 

81-90 

13 

0 

1 

16 

4 

7 

71-80 

13 

0 

1 

10 

4 

6 

61-70 

9 

0 

4 

12 

6 

2 

51-60 

15 

0 

1 

14 

5 

3 

41-50 

5 

1 

3 

7 

4 

0 

Missing 

5 

0 

1 

2 

0 

0 

Percent  of  response* 
(n-118) 

1 

9 

52 

22 

16 

TABLE  B-2 

Overall  Ability  to  Perform  versus  Mechanical  Aptitude 

Mechanical  Aptitude 
Score 

Missing 

Unsatis- 

factory 

Marginal  Satisfactory 

Very  satis- 
factory 

Excellent 

91-100 

3 

0 

1 

2 

2 

0 

81-90 

6 

0 

2 

11 

4 

6 

71-80 

4 

0 

1 

7 

4 

2 

61-70 

9 

0 

2 

6 

2 

3 

51-60 

6 

0 

0 

4 

3 

4 

41-50 

8 

0 

1 

6 

1 

1 

31-40 

10 

0 

1 

13 

2 

1 

21-30 

7 

1 

2 

9 

6 

1 

11-20 

4 

0 

1 

0 

2 

1 

1-10 

3 

0 

0 

3 

0 

0 

Missing 

5 

0 

1 

2 

0 

0 

Percent  of  responses 
(n-118) 

1 

9 

52 

• 

22 

16 
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TABLE  B-3 

Overall  Ability  to  Perform  versus  Administrative  Aptitude 

Administra- 
tive Aptitude 
Score 

Missing 

Unsatis- 

factory 

Marginal 

Satis- 

factory 

Very  Satis- 
factory 

Excellent 

91-100 

4 

0 

2 

3 

1 

i 

81-90 

10 

0 

0 

3 

6 

3 

71-80 

8 

0 

2 

8 

2 

3 

61-70 

8 

0 

1 

17 

4 

1 

51-60 

12 

0 

1 

11 

8 

5 

41-50 

6 

0 

1 

9 

2 

1 

31-40 

5 

1 

3 

5 

3 

3 

21-30 

6 

0 

0 

4 

0 

2 

11-20 

1 

0 

0 

0 

0 

0 

1-10 

0 

0 

1 

1 

0 

0 

Missing 

5 

G 

1 

2 

0 

0 

Percent  of 

Responses  (n—1 18) 

1 

9 

52 

22 

16 

TABLE  B-4 

Overall  Ability  to  Perform  versus  AFQT 


AFQT  Score 

Missing 

Unsatis- 

factory 

Marginal 

Satis- 

factory 

Very  Satis- 
factory 

Excellent 

91-100 

4 

0 

0 

5 

2 

i 

81-90 

4 

0 

2 

4 

1 

3 

71-80 

12 

0 

5 

6 

3 

10 

61-70 

9 

0 

0 

23 

11 

2 

51-60 

15 

0 

1 

10 

2 

1 

41-50 

13 

1 

3 

10 

6 

1 

31-40 

0 

0 

0 

0 

1 

0 

Missing 

8 

0 

1 

5 

0 

1 

Percent  of 

Responses  (n«114) 

1 

10 

51 

23 

16 
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TABLE  B-5 


Supervisor  Reactions  to  Basic  Questionnaires 


Clear 

Not  Clear  No  Response 

Clarity  of  instructions 

114 

6 

2 

Clarity  of  scales 

116 

3 

3 

TABLE  B-6 

Reported  Time  for  Supervisors  to  Complete 

Basic  Proficiency  Questionnaire 

Minutes 

Frequency 

Percent  of  those 
responding  (nM120) 

Not  reported 

2 

— 

10  or  less 

6 

5 

15 

1 

1 

20 

6 

5 

25 

4 

3 

30 

24 

20 

35 

4 

3 

40 

9 

8 

45 

26 

22 

50 

4 

3 

55 

1 

1 

60 

16 

13 

70 

1 

1 

80 

1 

1 

90 

11 

9 

Over  90 

6 

5 

Average  response  time  of  120  respondents  * 47.0  minutes 


37  of  120  respondents  (31%)  had  completed  one  or  more  training 
Training  Evaluation  questionnaires  previously. 
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TABLE  B-7 

Supervisor  Preference  for  Use  of  Separate  Answer  Sheet 

Preference 

Frequency 

Much  prefer  separate  answer  sheets 

19 

Somewhat  prefer  separate  answer  sheets 

14 

Makes  no  difference 

64 

/ 

Somewhat  prefer  responding  on  questionnaires 

13 

Much  prefer  responding  on  questionnaires 

9 

No  response 

3 

TABLE  B-8 

Graduate  Reactions  to  Questionnaire  Format 

Clear 

Not  Clear 

No  Response 

Clarity  of  instructions 

82 

13 

35 

Clarity  of  scales 

90 

6 

34 

TABLE  B-9 

Reported  Time  for  Graduates  to  Complete  Questionnaire 

Minutes 

Frequency 

Percent  of  those 
responding  (n=94) 

10  or  less 

5 

5 

15 

17 

18 

20 

15 

16 

25 

2 

2 

30 

26 

28 

35 

4 

4 

40 

3 

3 

45 

8 

9 

50 

2 

2 

60 

8 

9 

70 

1 

1 

90 

3 

3 

No  response 

34 

- 

Average  response  time  of  94  respondents  ■ 

31.6  minutes 

TABLE  B-10 

Graduate  Preference  for  use  of  Separate  Answer  Sheets 

Preference 

Frequency 

Much  prefer  separate  answer  sheets 

19 

Somewhat  prefer  separate  answer  sheets 

16 

Makes  no  difference 

52 

Somewhat  prefer  responding  on  questionnaires 

5 

Much  prefer  responding  on  questionnaires 

4 

No  response 

34 

I 

I 


TABLE  B-1 1 

Supervisor  Reaction  to  Task  Criticality  Questionnaire 

Supplement  A 

Clear 

Not  Clear 

No  Response 

Clarity  of  instructions 

45 

0 

0 

Clarity  of  scales 

45 

0 

0 

TABLE  B-1 2 

Reported  Time  for  Supervisors  to  Make 
Task  Criticality  Ratings 


Minutes 

Frequency 

Percent  of  those 
responding  (n=34) 

1 0 or  less 

5 

11 

15 

8 

18 

20 

8 

18 

25 

4 

9 

30 

9 

20 

45 

5 

11 

50 

1 

2 

60 

3 

7 

Over  90 

2 

4 

Average  response  time  for  34  respondents 

“ 30.1  minutes 

TABLE  B-1 3 

Supervisor  Confidence  in  Making 
Task  Criticality  Ratings 


Feelings  About  Making  Ratings  Frequency 


1 

1 

Comfortable 

35 

! 

Little  uneasy 

8 

t 

Very  uneasy 

2 

> » 

ii 

Extremely  uneasy 

0 
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TABLE  B-14 

Supervisor  Reaction  to  Training  Level  Standards 

Questionnaire— Supplement  B 

Clear 

Not  Clear 

No  Response 

Clarity  of  instructions 

41 

1 

0 

Clarity  of  scales 

41 

1 ' 

0 

TABLE  B-1 5 

Reported  Time  for  Supervisors  to  Make 

Training  Level  Standards  Ratings 

Minutes 

Frequency 

Percent  of  those 
responding  (n-40) 

Not  reported 

2 

— 

5 

2 

5 

10 

5 

13 

15 

9 

23 

20 

4 

10 

25 

1 

3 

30 

9 

23 

40 

4 

10 

45 

2 

5 

50 

2 

5 

55 

1 

3 

60 

1 

3 

Average  response  time  of  40  respondents  “ 2S.9  minutes 

TABLE  B-1 6 

Supervisor  Confidence  in 

Making  Training  Standards  Ratings 

Feelings  About  Making  Ratings 

Frequency 

Comfortable 

32 

Little  uneasy 

5 

Very  uneasy 

4 

Extremely  uneasy 

1 
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TABLE  B-17 

Supervisor  Reaction  to  Training  Deficiency  Impact  . 
Questionnaire— Supplement  C 


Clear 

Not  Clear 

No  Response 

Clarity  of  instructions 

33 

i 

0 

Clarity  of  scales 

34 

0 

0 

TABLE  B-18 

Reported  Time  for  Supervisors  to  Make 
Training  Deficiency  Impact  Ratings 


Minutes 

Frequency 

Percent  of  those 
responding  (n=34l 

10  or  less 

3 

9 

15 

6 

18 

17 

1 

3 

20 

6 

18 

25 

1 

3 

30 

14 

41 

40 

1 

3 

45 

1 

3 

Over  90 

1 

3 

Average  response  time  of  34  respondents  - 26.1  minutes 


TABLE  B-19 

Supervisor  Confidence  in  Making  Training  Deficiency 
Impact  Ratings 


Feelings  About  Making  Ratings  Frequency 


Comfortable 


25 

5 

4 


. 


'■ 


Little  uneasy 
Very  uneasy 
Extremely  uneasy 
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TABLE  B-20 

Task  Performance  by  Type  of  Task  on  which  Most  Time  was  Spent 

Based  on  reports  of  122  Supervisors;  cell  entries  are  percent  of  total  sample 

Type  of  Tasks 

Percent  of 

Total  Sample 
Reporting 

Task  was 
Performed 

Task  Number  Battery 

Maintenance 

(n-26,%»21) 

Troubleshoot-  Checking 

ing  on  Aircraft  in  Shop 

(n-66,  %-54l  (n-15,  %-t2) 

Troubleshoot- 
ing in  Shop 
<n-15.%-12) 

1 

22 

54 

12 

13 

98 

2 

22 

54 

12 

13 

98 

3 

17 

59 

11 

14 

89 

4 

21 

54 

13 

13 

90 

5 

14 

60 

14 

13 

77 

6 

20 

53 

13 

13 

92 

7 

14 

58 

12 

15 

75 

8 

16 

54 

12 

18 

47 

9 

19 

54 

14 

13 

89 

10 

21 

54 

13 

13 

98 

11 

21 

54 

12 

13 

94 

12 

19 

52 

14 

15 

60 

13 

16 

57 

74 

74 

84 

14 

15 

58 

13 

14 

89 

15 

16 

58 

13 

14 

91 

16 

18 

56 

13 

13 

93 

17 

19 

55 

13 

13 

95 

18 

12 

54 

18 

16 

41 

19 

8 

59 

16 

16 

40 

20 

21 

55 

13 

13 

92 

21 

21 

53 

13 

13 

84 

22 

9 

64 

18 

9 

9 

23 

0 

63 

25 

13 

6 

24 

25 

25 

25 

25 

7 

25 

29 

29 

29 

14 

6 

26 

20 

33 

33 

13 

12 

27 

22 

39 

26 

13 

19 

28 

60 

20 

20 

0 

4 

29 

25 

63 

13 

0 

7 

30 

23 

40 

27 

10 

25 

31 

18 

51 

18 

13 

32 

32 

29 

37 

17 

17 

52 

33 

30 

34 

19 

17 

52 

34 

36 

36 

13 

15 

50 

35 

35 

37 

13 

15 

51 

36 

34 

37 

16 

13 

62 

37 

34 

38 

14 

14 

53 

38 

36 

36 

14 

14 

59 

39 

38 

35 

13 

14 

57 

40 

5 

79 

5 

11 

16 

41 

5 

73 

9 

14 

18 

continued 
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TABLE  B- 20  (continued) 


Task  Number 

Type  of  Tasks 

Percent  of 

Total  Sample 
Reporting 

Task  was 
Performed 

Battery 
Maintenance 
(n=26,  %=21) 

Troubleshoot- 
ing on  Aircraft 
<n=66,  %*54) 

Checking 
in  Shop 
(n»15.  %=1 2) 

Troubleshoot- 
ing in  Shop 
<n*10.  %*12) 

42 

5 

74 

1 1 

1 1 

16 

43 

1 1 

69 

8 

1 1 

51 

44 

10 

72 

8 

10 

50 

45 

11 

70 

9 

1 1 

47 

46 

13 

69 

5 

12 

61 

47 

1 1 

69 

6 

14 

59 

48 

11 

68 

6 

14 

52 

49 

9 

74 

» 6 

1 1 

44 

50 

9 

74 

6 

1 1 

44 

51 

7 

73 

7 

13 

37 

52 

15 

84 

0 

0 

1 1 

53 

15 

85 

0 

0 

1 1 

54 

10 

90 

0 

0 

8 

55 

13 

66 

6 

15 

67 

56 

13 

65 

8 

14 

70 

57 

13 

66 

7 

13 

68 

58 

5 

71 

5 

18 

31 

59 

4 

71 

6 

19 

39 

60 

5 

70 

5 

20 

33 

61 

0 

77 

6 

17 

29 

62 

0 

74 

6 

20 

29 

63 

0 

74 

6 

21 

28 

64 

8 

70 

5 

16 

30 

65 

8 

71 

5 

16 

31 

66 

5 

73 

5 

16 

30 

67 

9 

68 

6 

16 

65 

68 

10 

66 

8 

16 

66 

69 

7 

70 

7 

16 

57 

70 

10 

68 

5 

17 

48 

71 

12 

68 

5 

15 

48 

72 

8 

69 

4 

19 

39 

73 

6 

59 

6 

29 

14 

74 

6 

56 

6 

31 

13 

75 

7 

50 

7 

36 

1 1 

76 

6 

72 

4 

17 

38 

77 

4 

73 

4 

18 

37 

78 

5 

73 

5 

18 

33 

79 

10 

60 

1C 

20 

33 

80 

10 

64 

8 

18 

41 

81 

10 

67 

8 

16 

42 
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TABLE  B-21 

Graduate  Reports  of  Training  Adequacy  by  Job  Task 


Job  Task 

Adequacy  for  Task  Performance 

Number 

NA 

1 

2 

3 

Average 

Safety 

i 

10 

1 

87 

32 

2.3 

2 

7 

6 

84 

33 

2.2 

3 

20 

5 

81 

24 

2.2 

4 

22 

11 

69 

28 

2.2 

Publica- 

5 

13 

15 

88 

14 

2.0 

tions 

6 

11 

15 

81 

23 

2.1 

7 

27 

20 

69 

14 

1.9 

8 

30 

28 

59 

13 

1.9 

Forms 

9 

14 

12 

80 

24 

2.1 

Funda- 

10 

8 

8 

83 

31 

2.2 

mentals 

11 

8 

7 

84 

31 

2.2 

12 

25 

13 

67 

25 

2.1 

13 

11 

13 

81 

25 

2.1 

14 

8 

10 

90 

22 

2.1 

15 

9 

11 

90 

20 

2.1 

16 

5 

15 

87 

23 

2.1 

17 

7 

13 

88 

22 

2.1 

Meters/ 

18 

55 

20 

44 

11 

1.9 

Testers 

19 

49 

20 

51 

10 

1.9 

20 

8 

7 

82 

33 

2.2 

21 

12 

8 

79 

31 

2.2 

22 

83 

17 

22 

8 

1.8 

23 

87 

19 

17 

7 

1.7 

24 

74 

15 

34 

7 

1.9 

25 

82 

15 

25 

8 

1.9 

Test 

26 

66 

19 

37 

8 

1.8 

Stands/ 

27 

63 

18 

41 

8 

1.9 

Load 

Banks 

28 

77 

19 

27 

7 

1.8 

29 

77 

21 

26 

6 

1.7 

30 

53 

26 

43 

8 

1.8 

31 

46 

30 

47 

7 

1.7 

Battery 

32 

55 

9 

45 

21 

2.2 

Shop 

33 

49 

7 

51 

23 

2.2 

34 

48 

12 

46 

24 

2.1 

35 

49 

11 

47 

23 

2.1 

36 

39 

11 

53 

27 

2.2 

37 

42 

14 

53 

21 

2.1 

38 

36 

9 

63 

22 

271 

39 

37 

14 

56 

23 

2.1 

Adequacy  for  CDC  Work 
NA  1 2 3 Average 


26  3 80  21  2.2 

29  4 75  22  2.2 

30  7 72  21  2.1 

36  8 64  22  2. 1 


Adequacy  for  FTD  Training 


16 

40 

9 

1.9 

89 

6 

27 

15 

42 

11 

1.9 

89 

5 

28 

16 

40 

7 

1.9 

5 

19 

21 

32 

9 

1.8 

104 

7 

13 

10 

53 

13 

2.0 

101 

4 

18 

11 

58 

16 

2.1 

102 

2 

16 

12 

59 

15 

2.0 

101 

3 

16 

19 

49 

12 

1.9 

107 

6 

10 

12 

53 

13 

2.0 

104 

6 

13 

14 

61 

10 

2.0 

103 

5 

14 

17 

64 

11 

1.9 

99 

6 

17 

11 

58 

10 

2.0 

106 

3 

13 

12 

56 

8 

1.9 

105 

3 

15 

18 

43 

6 

1.8 

110 

5 

11 

18 

51 

8 

1.9 

110 

6 

9 

10 

59 

16 

2.1 

5 

16 

13 

59 

18 

2.1 

103 

5 

15 

26 

33 

8 

1.7 

114 

5 

9 

28 

29 

7 

1.7 

117 

4 

7 

20 

36 

6 

1.8 

114 

5 

9 

21 

34 

4 

1.7 

117 

4 

7 

46 
46 
41 
38 

23  49 

58  19  49 


35  10  73 

38  9 72 

37  13  67 

41  13  66 

37  9 72 

42  10  67 


10  2.1 
14  2.1 

12  2.0 


13  2.0 

10  2.0 
12  2.0 
11  2.0 


9 

2.2 

8 

2.2 

0 

2.2 

0 

2.2 

Job  Task 
Number 


A/C  Power  40 
Systems  41 


Adequacy  for  Task  Performance 
NA  1 2 3 Average 


Adequacy  for  CDC  Work 
NA  1 2 3 Average 


Adequacy  for  FTD  Training 
IA  1 2 3 Average 


AC  Control/  55 
Warning  ra 

Systems  R7 


104 

4 

13 

4 

2.0 

99 

4 

22 

5 

2.0 

101 

3 

22 

4 

2.0 

90 

5 

29 

6 

2.0  | 

94 

3 

28 

5 

2.1 

91 

4 

30 

5 

2.0 

93 

6 

27 

4 

1.9 

93 

6 

27 

4 

1.9 

96 

6 

24 

4 

1.9 

96 

10 

21 

3 

1.8 

93 

9 

24 

4 

19  1 

99 

7 

21 

3 

1.9 

106 

8 

13 

3 

1.8 

104 

7 

16 

3 

1.8 

105 

7 

15 

3 

1.3 

1 * Somewhat  less  than  adequate.  2 » Adequate;  3 - Somewhat  more  than  necessary 
Average  score  based  only  on  graduates  reporting  involvement  in  the  task 


TABLE  B-22 

Distribution  of  Performance  Ratings* 


Frequency  of  Response 

Level  of  Proficiency  Percent  Rated96 

Not  Cap- Very  Partly  Compe-  Highly  proficient  Proficiency 

able  Limited  Proficient  tent  Competent  or  getter  

(0)  (1)  (2)  (3)  (4)  N Mean 


1 

3a 

1 

2 

2 

1 

21 

78 

19 

99 

119 

2.97 

2 

3b 

2 

2 

2 

- 

2 

14 

73 

29 

98 

118 

3.09 

3 

3d  5 

1 

13 

11 

— 

1 

19 

65 

23 

99 

108 

3.02 

4 

3f 

8 

6 

5 

- 

3 

15 

66 

24 

97 

108 

3.03 

5 

4a 

2 

28 

23 

— 

9 

35 

41 

7 

90 

92 

2.50 

6 

4b 

1 

9 

7 

- 

9 

41 

49 

13 

92 

1 12 

2.59 

7 

4d 

3 

30 

25 

— 

5 

31 

43 

10 

94 

89 

2.65 

8 

4e 

2 

64 

52 

- 

9 

18 

24 

5 

84 

56 

2.45 

9 

7d 

5 

9 

7 

— 

8 

34 

52 

14 

93 

108 

2.67 

10 

9a 

4 

3 

2 

- 

3 

20 

68 

24 

97 

115 

2.98 

11 

9a 

7 

3 

2 

— 

2 

15 

71 

24 

98 

112 

3.04 

12 

9b 

3 

47 

39 

1 

5 

21 

37 

8 

92 

72 

2.64 

13 

9c 

1 

19 

16 

- 

3 

20 

60 

19 

97 

102 

2.93 

14 

9d 

1 

14 

11 

- 

5 

28 

58 

16 

98 

107 

2.79 

15 

9g 

2 

11 

9 

— 

4 

20 

65 

20 

96 

109 

2.93 

16 

9h 

3 

8 

7 

1 

6 

24 

63 

17 

94 

111 

2.80 

17 

9h 

2 

6 

5 

- 

3 

23 

66 

22 

97 

114 

2.94 

18 

13al 

5 

69 

57 

1 

5 

20 

19 

3 

88 

48 

2.38 

19 

13al 

1 

73 

60 

1 

6 

17 

19 

5 

85 

48 

2.44 

20 

13a2 

4 

10 

8 

1 

5 

22 

62 

19 

95 

108 

2.88 

21 

13a2 

3 

17 

14 

— 

6 

19 

59 

18 

94 

102 

2.87 

22 

13a3 

2 

110 

90 

— 

1 

3 

3 

3 

90 

10 

2.80 

23 

13a3 

2 

113 

93 

— 

1 

2 

1 

3 

86 

7 

2.86 

24 

13a5 

1 

114 

93 

1 

— 

2 

4 

— 

86 

7 

2.50 

25 

13a5 

2 

114 

93 

— 

- 

3 

3 

- 

100 

6 

2.50 

26 

14a2 

6 

102 

84 

- 

3 

4 

6 

1 

79 

14 

2.36 

27 

14a2 

1 

98 

80 

— 

9 

5 

9 

— 

61 

23 

2.00 

28 

14c2 

1 

116 

95 

- 

2 

— 

2 

1 

60 

5 

2.40 

29 

14c2 

2 

112 

92 

— 

6 

— 

2 

— 

25 

8 

2.50 

30 

14d2 

1 

91 

75 

- 

6 

13 

9 

2 

80 

30 

2.33 

31 

14d2 

1 

82 

67 

— 

12 

12 

14 

1 

69 

39 

2.10 

32 

1 5a  1 

6 

54 

54 

— 

2 

18 

29 

13 

96 

62 

2.85 

33 

15a1 

2 

57 

47 

- 

3 

18 

30 

12 

95 

63 

2.81 

34 

15a2 

1 

60 

49 

- 

2 

15 

33 

11 

97 

61 

2.87 

35 

15a2 

1 

59 

48 

- 

2 

16 

33 

11 

97 

62 

2.85 

36 

15b2 

1 

45 

37 

- 

1 

19 

43 

13 

99 

76 

2.89 

37 

15b3 

1 

56 

46 

— 

4 

16 

35 

10 

94 

65 

2.78 

38 

1 5c  1 

1 

49 

40 

— 

1 

16 

42 

13 

99 

72 

2.93 

39 

15c2 

1 

52 

43 

- 

2 

15 

39 

13 

97 

69 

2.91 

40 

16a2 

3 

101 

83 

- 

4 

7 

7 

- 

78 

18 

2.17 

41 

16a2 

3 

98 

80 

— 

3 

9 

9 

— 

86 

21 

2.29 

continued 


Task  Not 

Job  Task  STS  Item  Data  Performed 
Number  Number  Missing 


N % 


Based  on  122  Supervisor  reports 
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TABLE  B- 22  (continued) 


Frequency  of  Response 

Level  of  Proficiency  nt  Average 

Part,V  Rated 

Not  Cap- Very  Partly  Compe- Highly  or°8etter'  Proficiency 

able  Limited  Proficient  tent  Competent  

(0)  (1)  (2)  (3)  (4)  N Mean 


42 

16a2 

1 

102 

84 

1 

4 

8 

5 

1 

74 

19 

2.05 

43 

16b2 

- 

60 

49 

1 

18 

20 

19 

4 

69 

62 

2.11 

44 

16b2 

- 

61 

50 

1 

15 

21 

20 

4 

74 

61 

2.18 

45 

16b2 

- 

65 

53 

5 

13 

20 

14 

5 

68 

57 

2.02 

46 

16c2 

1 

47 

39 

1 

7 

25 

35 

6 

89 

74 

2.51 

47 

16c2 

1 

50 

41 

1 

7 

24 

33 

6 

89 

71 

2.51 

48 

16c2 

- 

59 

48 

2 

13 

19 

24 

5 

76 

63 

2.27 

49 

16d2 

- 

67 

55 

1 

7 

17 

25 

5 

87 

54 

2 52 

50 

16d2 

1 

67 

55 

- 

7 

17 

25 

5 

87 

54 

2.52 

51 

16d2 

1 

76 

62 

- 

7 

15 

19 

4 

84 

45 

2.44 

52 

16  e2a 

- 

109 

89 

1 

3 

5 

4 

— 

69 

13 

1.92 

53 

16e2a 

- 

109 

89 

1 

2 

5 

5 

— 

77 

13 

2.08 

54 

16e2a 

1 

111 

91 

2 

3 

2 

3 

— 

50 

10 

1.60 

55 

1 7a2 

10 

32 

26 

- 

3 

24 

42 

11 

96 

80 

2.76 

56 

1 7a2 

1 

36 

30 

- 

2 

26 

46 

11 

98 

85 

2.78 

57 

17a2 

2 

38 

31 

1 

9 

29 

34 

9 

88 

82 

2.50 

58 

17b2 

- 

84 

69 

- 

6 

13 

16 

3 

84 

38 

2.42 

59 

1 7b2 

- 

74 

61 

- 

8 

16 

20 

4 

83 

48 

2.42 

60 

17b2 

- 

82 

67 

1 

13 

9 

13 

4 

65 

40 

2.15 

61 

1 7e2a 

1 

87 

71 

1 

8 

9 

13 

3 

74 

34 

2.26 

62 

1 7e2a 

1 

87 

71 

1 

6 

9 

15 

3 

79 

34 

2.38 

63 

1 7e2a 

1 

88 

72 

1 

9 

10 

10 

3 

70 

33 

2.15 

64 

17e2b 

- 

85 

70 

1 

4 

9 

19 

4 

86 

37 

2.57 

65 

1 7e2b 

- 

84 

69 

1 

3’ 

11 

19 

4 

89 

38 

2.58 

66 

1 7e2b 

- 

85 

70 

2 

7 

12 

13 

3 

76 

37 

2.22 

67 

1 7 12a 

- 

43 

35 

- 

6 

16 

47 

10 

92 

79 

2.77 

68 

1 7f2a 

- 

42 

34 

- 

5 

21 

43 

11 

94 

80 

2.75 

69 

17f2a 

3 

52 

43 

2 

7 

18 

31 

9 

87 

67 

2.57 

70 

17f2b 

1 

63 

52 

- 

7 

14 

27 

10 

88 

58 

2.69 

71 

17f2b 

1 

64 

52 

- 

7 

15 

26 

9 

88 

57 

2.65 

72 

17f2b 

2 

74 

61 

1 

4 

14 

19 

8 

89 

46 

2.63 

73 

17f2c 

- 

105 

86 

- 

3 

4 

8 

2 

82 

17 

2.52 

74 

17f2c 

- 

106 

87 

— 

3 

4 

8 

1 

81 

16 

2.44 

75 

1 7f2c 

- 

108 

89 

- 

2 

3 

6 

3 

86 

14 

2.71 

76 

17f2d 

- 

75 

61 

- 

5 

17 

19 

6 

89 

47 

2.55 

77 

17f2d 

— 

77 

63 

- 

6 

17 

18 

4 

37 

45 

2.44 

78 

17f2d 

- 

82 

67 

1 

8 

16 

12 

3 

78 

40 

2.20 

79 

1 7h2b 

1 

81 

66 

1 

6 

11 

18 

4 

83 

40 

2.45 

80 

17h2b 

- 

72 

59 

1 

6 

16 

23 

4 

86 

50 

2.46 

81 

17h2b 

— 

71 

58 

2 

9 

19 

17 

4' 

78 

51 

2.24 

Job  Task  STS  Item  Data  Task  Not 

Number  Number  Missing  Performed 


N % 
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TABLE  B-23 

Supervisor-Graduate  Agreement  on  Which  Tasks  Were  Performed 


Agreement Disagreement 


Job  Task  Numb 

STS  Number 

Data  for  both  N 

Both  Yes 

Both  No 

Total 

Supervisor  Yes, 
Graduate  No 

Supervisor  No. 
Graduate  Yes 

Total 

Percent 

Supervisor  Yes 

Percent 
Graduate  Yes 

1 

3a 

93 

90 

0 

90 

8 

2 

10 

98 

92 

2 

3b 

93 

94 

0 

94 

4 

2 

6 

98 

96 

3 

3d  5 

94 

79 

3 

82 

12 

6 

18 

91 

85 

4 

3f 

90 

78 

2 

80 

18 

2 

20 

96 

80 

5 

4 a 

94 

68 

1 

69 

7 

24 

31 

75 

92 

6 

4b 

94 

83 

0 

83 

9 

8 

17 

92 

91 

7 

4d 

92 

64 

8 

72 

13 

15 

28 

77 

79 

8 

4e 

93 

33 

12 

45 

12 

43 

55 

45 

77 

9 

7d 

92 

83 

2 

85 

11 

4 

15 

94 

87 

10 

9a 

94 

90 

0 

90 

6 

4 

10 

96 

94 

11 

9a 

89 

91 

1 

92 

6 

2 

8 

97 

93 

12 

9b 

93 

54 

6 

60 

12 

28 

40 

66 

82 

13 

9c 

94 

82 

3 

85 

5 

10 

15 

87 

92 

14 

9d 

94 

84 

0 

84 

5 

11 

16 

89 

95 

15 

9g 

94 

84 

0 

84 

9 

7 

16 

93 

91 

16 

9h 

94 

89 

1 

90 

4 

6 

10 

93 

95 

17 

9h 

94 

90 

0 

90 

5 

5 

10 

95 

95 

18 

1 3a  1 

90 

34 

31 

65 

9 

26 

35 

43 

60 

19 

1 3a  1 

93 

34 

29 

63 

8 

29 

37 

42 

63 

20 

13a2 

93 

87 

1 

88 

6 

6 

12 

93 

93 

21 

13a2 

92 

79 

2 

81 

5 

14 

19 

84 

93 

22 

13a3 

92 

7 

63 

70 

3 

27 

30 

10 

34 

23 

13a3 

92 

3 

65 

68 

4 

28 

32 

7 

31 

24 

13a5 

93 

6 

54 

60 

2 

38 

40 

8 

44 

25 

13a5 

93 

4 

62 

66 

3 

31 

34 

7 

35 

26 

14a2 

89 

9 

52 

61 

4 

35 

39 

13 

44 

27 

14a2 

93 

13 

44 

57 

9 

34 

43 

22 

47 

28 

14c2 

93 

1 

60 

61 

3 

36 

39 

4 

37 

29 

14c2 

92 

4 

59 

63 

4 

33 

37 

8 

37 

30 

■ 14d2 

93 

20 

34 

54 

9 

37 

46 

29 

57 

31 

14d2 

93 

27 

25 

52 

9 

39 

48 

36 

63 

32 

15a1 

89 

44 

31 

75 

12 

13 

25 

56 

57 

33 

1 5a1 

93 

45 

31 

76 

10 

14 

24 

55 

59 

34 

1 5a2 

93 

43 

33 

76 

8 

16 

24 

51 

59 

35 

15a2 

93 

42 

32 

74 

10 

16 

26 

52 

58 

36 

1 5b2 

93 

56 

26 

82 

8 

10 

18 

64 

66 

37 

1 5b3 

93 

46 

27 

73 

8 

19 

27 

54 

65 

38 

15c1 

93 

55 

25 

80 

4 

16 

20 

59 

71 

39 

15c2 

93 

52 

25 

77 

4 

19 

23 

56 

71 

40 

16a2 

92 

4 

51 

55 

9 

36 

45 

13 

. 40 

41 

16a2 

92 

11 

43 

54 

7 

39 

46 

18 

50 

continued 
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TABLE  B-23  (continued) 


Job  Task  Number 

STS  Number 

Data  for  both  N 

Agreement 

$ o 

> 2 

£L 

O o 

CO  CO 

Total 

Supervisor  Yes. 
Graduate  No 

O 

f 

Supervisor  No,  j 
Graduate  Yes  | 

S 

3 

Total 

Percent 

Supervisor  Yes 

Percent 

Graduate  Yes 

42 

16a2 

94 

11 

47 

58 

5 

37 

42 

16 

48 

43 

16b2 

94 

39 

22 

61 

13 

26 

39 

52 

65 

44 

16b2 

94 

41 

19 

60 

14 

26 

40 

55 

67 

45 

16b2 

94 

38 

22 

60 

12 

28 

40 

50 

66 

46 

16c2 

94 

47 

20 

67 

16 

17 

33 

63 

64 

47 

16c2 

94 

46 

18 

64 

18 

18 

36 

64 

64 

48 

16c2 

94 

39 

24 

63 

14 

23 

37 

53 

62 

49 

16d2 

93 

29 

30 

59 

17 

24 

41 

46 

53 

50 

16d2 

93 

35 

30 

65 

12 

23 

35 

47 

58 

51 

16d2 

93 

26 

34 

60 

14 

26 

40 

40 

52 

52 

16e2 

94 

6 

57 

63 

4 

33 

37 

10 

39 

53 

16e2 

94 

7 

54 

61 

4 

35 

39 

11 

42 

54 

16e2 

93 

6 

55 

61 

2 

37 

39 

8 

43 

55 

1 7a2 

88 

50 

14 

64 

23 

13 

34 

73 

63 

56 

17a2 

94 

59 

14 

73 

13 

14 

27 

72 

73 

57 

1 7a2 

93 

55 

14 

69 

14 

17 

31 

69 

72 

58 

1 7b2 

94 

19 

34 

53 

10 

37 

47 

29 

56 

59 

1 7b2 

94 

28 

22 

50 

14 

36 

50 

42 

64 

60 

1 7b2 

94 

24 

33 

57 

8 

35 

43 

32 

59 

61 

17e2a 

94 

20 

43 

63 

7 

30 

37 

27 

50 

62 

1 7e2a 

93 

19 

43 

62 

8 

30 

38 

27 

49 

63 

1 7e2a 

94 

16 

43 

59 

10 

31 

41 

26 

47 

64 

1 7e2b 

94 

18 

40 

58 

11 

31 

42 

29 

49 

65 

17e2b 

94 

18 

40 

58 

12 

30 

42 

30 

48 

66 

1 7e2b 

94 

17 

44 

61 

10 

29 

39 

27 

46 

67 

17f2a 

93 

56 

18 

74 

13 

13 

26 

69 

69 

68 

1 7f2a 

94 

55 

18 

73 

13 

14 

27 

68 

69 

69 

1 7f2a 

92 

46 

20 

66 

14 

20 

34 

60 

66 

70 

17f2b 

94 

38 

30 

68 

12 

20 

32 

50 

58 

71 

17f2b 

94 

37 

27 

64 

11 

25 

36 

48 

62 

72 

1 7f2b 

94 

28 

30 

58 

12 

30 

42 

40 

58 

73 

1 7f2c 

94 

12 

49 

61 

3 

36 

39 

15 

48 

74 

17f2c 

94 

11 

48 

59 

3 

38 

41 

12 

47 

75 

17f2c 

94 

9 

50 

59 

3 

38 

41 

12 

47 

76 

17f2d 

94 

26 

33 

59 

14 

27 

41 

40 

53 

77 

1 7f2d 

94 

26 

34 

60 

13 

27 

40 

39 

53 

78 

17f2d 

94 

21 

40 

61 

11 

28 

39 

32 

49 

79 

1 7h2b 

93 

14 

46 

60 

16 

24 

40 

30 

38 

80 

17h2b 

94 

21 

39 

60 

18 

22 

40 

39 

43 

81 

17h2b 

94 

21 

41 

62 

18 

20 

38 

39 

41 
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TABLE  B-24 

Reported  Reasons  for  Nonperformance  of  Tasks 

Based  on  1 22  Supervisor  Reports 

Job  Task 
Number 

Number 
Not  Per- 
forming 

Not  Yet 
But  Will 
Be  Re- 
quired 

Not  Rel- 
evant 

At  Work 
Site 

Not 

Allowed 

At  Work 
Site 

Not 

Required 

Hard- 

ware 

Not  Re- 
quired 

Other 

i 

2 

2 

— 

— 

— 

— 

2 

2 

2 

- 

- 

- 

- 

3 

13 

11 

2 

- 

- 

- 

4 

6 

6 

- 

- 

- 

- 

5 

28 

25 

1 

1 

- 

- 

6 

9 

8 

1 

- 

- 

- 

7 

30 

21 

7 

1 

1 

- 

8 

64 

54 

7 

1 

- 

- 

9 

9 

9 

1 

- 

- 

- 

10 

3 

3 

- 

- 

- 

- 

11 

3 

3 

- 

- 

- 

- 

12 

47 

35 

5 

- 

6 

2 

13 

19 

17 

2 

- 

- 

1 

14 

14 

13 

1 

- 

- 

- 

15 

11 

9 

2 

— 

- 

- 

16 

8 

6 

2 

- 

- 

- 

17 

6 

5 

1 

- 

- 

- 

18 

69 

54 

6 

1 

3 

2 

19 

73 

55 

7 

1 

4 

2 

20 

10 

7 

2 

- 

- 

- 

21 

17 

10 

5 

1 

- 

1 

22 

110 

58 

29 

2 

15 

3 

23 

113 

60 

30 

2 

16 

3 

24 

114 

56 

28 

4 

21 

3 

25 

114 

57 

28 

4 

21 

3 

26 

102 

67 

18 

4 

7 

4 

27 

98 

64 

18 

4 

7 

3 

28 

116 

38 

43 

6 

24 

4 

29 

112 

33 

44 

6 

24 

4 

30 

91 

59 

18 

2 

7 

4 

31 

82 

53 

16 

2 

7 

4 

32 

54 

27 

16 

1 

4 

4 

33 

57 

30 

16 

1 

5 

4 

34 

60 

31 

20 

1 

4 

3 

35 

59 

30 

20 

1 

4 

3 

36 

45 

24 

13 

1 

3 

3 

37 

56 

34 

14 

1 

3 

3 

38 

49 

26 

14 

1 

4 

3 

39 

52 

29 

14 

1 

4 

3 

40 

101 

32 

46 

2 

18 

2 

41 

98 

29 

45 

— continued  - 

2 

18 

2 
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TABLE  B-24  (continued) 

Job  Task 
Number 

Number 
Not  Per 
forming 

Not  Vet 
But  Will 

Be  Re- 
quired 

Not  Rel- 
evant 

At  Work 
Site 

Not 

Allowed 

At  Work 
Site 

Not 

Required 

Hard- 

ware 

Not  Re- 
quired 

Other 

42 

102 

36 

47 

1 

17 

i 

43 

60 

50 

6 

— 

1 

i 

44 

61 

52 

6 

— 

1 

i 

45 

65 

55 

6 

1 

1 

i 

46 

47 

40 

6 

- 

- 

i 

47 

50 

43 

6 

— 

- — 

i 

48 

59 

52 

6 

- 

- 

i 

49 

67 

57 

7 

- 

2 

i 

50 

67 

57 

7 

- 

2 

i 

51 

76 

63 

9 

- 

2 

2 

52 

109 

35 

52 

1 

16 

4 

53 

109 

33 

53 

2 

16 

4 

54 

111 

36 

53 

2 

16 

3 

55 

32 

26 

5 

- 

— 

1 

56 

36 

28 

'7 

— 

— 

1 

57 

38 

31 

6 

- 

— 

1 

58 

84 

75 

8 

- 

- 

1 

59 

74 

64 

9 

- 

— 

1 

60 

82 

72 

9 

— 

— 

1 

61 

87 

71 

11 

1 

3 

1 

62 

87 

68 

14 

1 

3 

1 

63 

88 

71 

11 

1 

3 

1 

64 

85 

72 

10 

- 

2 

1 

65 

84 

71 

10 

- 

2 

1 

66 

85 

72 

10 

- 

2 

1 

67 

43 

35 

6 

- 

- 

1 

68 

42 

34 

6 

- 

— 

1 

69 

52 

43 

8 

- 

1 

1 

70 

63 

43 

14 

- 

4 

1 

71 

64 

43 

15 

- 

4 

M 

72 

74 

51 

17 

- 

4 

1 

73 

105 

51 

41 

- 

10 

1 

74 

106 

51 

42 

- 

10 

1 

75 

108 

54 

41 

- 

10 

1 

76 

75 

59 

13 

- 

2 

1 

77 

77 

61 

13 

- 

2 

1 

78 

82 

65 

14 

— 

2 

1 

79 

81 

58 

18 

3 

1 

1 

80 

72 

59 

11 

- 

1 

1 

81 

71 

58 

11 

- 

1 

1 
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TABLE  B-25 

Basis  for  Supervisor  Proficiency  Ratings 


Percent  of  Supervisors  Reporting 


Job  Task 
Number 

N 

Own  Task 
Observa- 
tions 

Other  Task 
Observa- 
tions 

General 

1 mpres- 
sions 

i 

116 

78 

16 

5 

2 

116 

85 

1 1 

3 

3 

105 

77 

18 

5 

4 

106 

79 

16 

5 

5 

89 

75 

18 

7 

6 

108 

78 

18 

5 

7 

93 

65 

24 

5 

8 

55 

69 

25 

5 

9 

106 

86 

11 

3 

10 

114 

82 

14 

4 

11 

110 

86 

9 

5 

12 

70 

80 

16 

4 

13 

99 

79 

19 

2 

14 

103 

79 

19 

2 

15 

105 

78 

18 

4 

16 

108 

78 

18 

5 

17 

111 

77 

20 

4 

18 

47 

66 

34 

0 

19 

47 

70 

30 

0 

20 

106 

76 

22 

2 

21 

101 

78 

19 

3 

22 

9 

78 

22 

0 

23 

6 

83 

17 

0 

24 

7 

57 

43 

0 

25 

6 

50 

50 

0 

26 

14 

64 

36 

0 

27 

22 

77 

23 

0 

28 

5 

80 

20 

0 

29 

8 

88 

12 

0 

30 

29 

66 

34 

0 

31 

38 

68 

32 

0 

32 

60 

62 

38 

0 

33 

62 

60 

40 

0 

34 

59 

84 

16 

0 

35 

60 

70 

30 

0 

36 

74 

69 

31 

0 

37 

64 

66 

34 

0 

38 

70 

67 

33 

0 

39 

67 

67 

33 

0 

40 

17 

88 

12 

0 

41 

20 

85 

15 

0 

continued 
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TABLE  B-25  (continued) 

Job  Task 
Number 

N 

Percent  of  Supervisors  Reporting 

Own  Task 
Observa- 
tions 

Other  Task 
Observa- 
tions 

General 

1 repres- 
sions 

42 

17 

71 

29 

0 

43 

60 

58 

42 

0 

44 

58 

59 

41 

0 

45 

54 

61 

39 

0 

46 

72 

60 

39 

1 

47 

68 

54 

46 

0 

48 

60 

58 

42 

0 

49 

49 

53 

47 

0 

50 

50 

54 

46 

0 

51 

43 

49 

51 

0 

52 

12 

83 

17 

0 

53 

12 

75 

25 

0 

54 

9 

67 

33 

0 

55 

76 

67 

30 

3 

56 

81 

65 

32 

3 

57 

78 

63 

33 

4 

58 

38 

66 

32 

2 

59 

48 

58 

42’ 

0 

60 

40 

63 

37 

0 

61 

34 

65 

35 

0 

62 

34 

68 

32 

0 

63 

33 

67 

33 

0 

64 

37 

68 

32 

0 

65 

38 

66 

34 

0 

66 

37 

70 

30 

0 

67 

77 

68 

30 

2 

68 

79 

67 

30 

3 

69 

65 

68 

28 

4 

70 

57 

67 

32 

1 

71 

56 

64 

34 

2 

72 

46 

70 

28 

2 

73 

17 

71 

29 

0 

74 

16 

63 

37 

0 

75 

14 

71 

29 

0 

76 

45 

60 

36 

4 

77 

43 

60 

35 

5 

78 

49 

47 

29 

4 

79 

38 

61 

39 

0 

80 

48 

58 

40 

2 

81 

49 

59 

39 

2 
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TABLE  B-26 

Task  Proficiency  Ratings  as  a Function  of  the  Basis 
for  the  Rating 


Basis 


Task 

Number  General  Impressions 


Own  Observations  Other  Observations 

on  Task  on  Task 


N 

Mean* 

N 

Mean* 

N 

Mean* 

1 

6 

4.17 

91 

3.98 

19 

3.90 

2 

4 

4.00 

98 

4.10 

13 

4.08 

3 

5 

3.80 

81 

4.07 

19 

3.84 

4 

5 

4.00 

83 

4.05 

17 

4.00 

5 

6 

3.50 

67 

3.45 

16 

3.75 

6 

5 

3.80 

84 

3.51 

19 

3.68 

7 

5 

3.40 

60 

3.63 

22 

3.78 

8 

3 

3.67 

38 

3.42 

14 

3.50 

9 

3 

3.67 

91 

3.68 

12 

3.50 

10 

4 

4.00 

92 

3.99 

16 

3.88 

11 

4 

4.50 

94 

4.02 

10 

3.90 

12 

2 

4.00 

56 

3.64 

11 

3.64 

13 

2 

4.00 

78 

3.92 

19 

3.90 

14 

2 

4.00 

81 

3.80 

20 

3.65 

15 

4 

3.75 

82 

3.94 

19 

3.84 

16 

5 

3.60 

84 

3.77 

19 

3.42 

17 

4 

3.75 

85 

3.97 

22 

3.82 

18 

0 

— 

31 

3.42 

16 

3.25 

19 

0 

- 

33 

3.42 

14 

3.43 

20 

2 

3.50 

81 

3.91 

23 

3.78 

21 

3 

3.33 

79 

3.91 

19 

3.79 

22 

0 

- 

7 

4.14 

2 

3.00 

23 

0 

— 

5 

4.20 

1 

3.00 

24 

0 

— 

4 

3.75 

3 

2.67 

25 

0 

- 

3 

3.33 

3 

3.67 

26 

0 

- 

9 

3.22 

5 

3.60 

27 

0 

- 

17 

2.88 

5 

3.20 

28 

0 

- 

4 

3.25 

1 

4.00 

29 

0 

- 

7 

2.87 

1 

4.00 

30 

0 

- 

19 

3.05 

10 

3.10 

31 

0 

— 

26 

3.12 

12 

3.17 

32 

0 

— 

37 

4.03 

23 

3.56 

33 

0 

- 

37 

3.95 

24 

3.58 

34 

0 

- 

42 

3.98 

17 

3.59 

35 

0 

— 

42 

3.98 

18 

3.56 

36 

0 

— 

51 

4.00 

23 

3.65 

37 

0 

— 

42 

3.86 

22 

3.64 

38 

0 

— 

47 

4.09 

23 

3.61 

39 

0 

— 

45 

4.04 

22 

3.64 

40 

0 

— 

14 

3.14 

2 

3.50 

41 

0 

- 

16 

— continued  - 

3.38 

3 

3.00 

152 


TABLE  B-26  (continued) 

Basis 

Task 

Number 

General  Impressions 

Own  Observations 
on  Task 

Other  Observations 
on  Task 

N 

Mean* 

N 

Mean* 

N 

Mean* 

42 

0 

— 

12 

3.00 

5 

2.80 

43 

0 

- 

35 

3.26 

25 

2.88 

44 

0 

- 

34 

3.18 

24 

3.21 

45 

0 

— 

33 

3.06 

21 

2.91 

46 

1 

3.00 

42 

3.71 

28 

3.25 

47 

0 

- 

37 

3.70 

31 

3.29 

48 

0 

— 

35 

3.34 

25 

3.12 

49 

0 

- 

26 

3.77  ' 

23 

3.35 

50 

0 

- 

27 

3.74 

23 

3.35 

51 

0 

- 

21 

3.71 

22 

3.16 

52 

0 

- 

10 

3.00 

2 

2.50 

53 

0 

- 

9 

3.11 

3 

3.00 

54 

0 

- 

6 

2.83 

3 

2.00 

55 

2 

3.50 

50 

3.84 

23 

3.61 

56 

2 

4.00 

53 

3.83 

26 

3.69 

57 

3 

3.67 

49 

3.57 

26 

3.35 

58 

1 

4.00 

25 

3.60 

12 

3.00 

59 

0 

- 

28 

3.57 

20 

3.20 

60 

0 

- 

25 

3.28 

15 

2.93 

61 

0 

- 

22 

3.50 

12 

2.83 

62 

0 

- 

23 

3.48 

11 

3.18 

63 

0 

— 

22 

3.27 

11 

2.91 

64 

0 

— 

25 

3.68 

12 

3.33 

65 

0 

- 

25 

3.68 

13 

3.39 

66 

0 

26 

3.27 

11 

3.09 

67 

2 

3.50 

52 

3.90 

23 

3.52 

68 

2 

3.50 

53 

3.83 

24 

3.58 

69 

3 

2.99 

44 

3.59 

18 

3.50 

70 

1 

4.00 

38 

3.79 

18 

3.44 

71 

1 

4.00 

36 

3.78 

19 

3.36 

72 

1 

4.00 

31 

3.81 

13 

3.15 

73 

0 

— 

12 

3.50 

5 

3.60 

74 

0 

— 

10 

3.30 

6 

3.67 

75 

0 

— 

10 

3.70 

4 

3.75 

76 

2 

3.50 

27 

3.63 

16 

3.38 

77 

2 

3.50 

26 

3.60 

15 

3.27 

78 

2 

3.50 

23 

3.30 

14 

3.07 

79 

0 

- 

23 

3.61 

15 

3.27 

80 

1 

2.00 

28 

3.64 

19 

3.26 

81 

1 

2.00 

29 

3.41 

19 

3.05 

All  Tasks 

35 

3.63 

81 

3.61 

81 

3.40 

• Because  of  the  manner  in  which  the  means  were  calculated.  1 must  be  subtracted  from  all 
of  them  to  make  them  comparable  with  other  ratings. 
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TABLE  B-27 

Training  Standard  Levels  for  Selected  STS  Job  Elements 


STS  Job 
Element 

Not 

Rated 

Present  Level 

Number 

Level 

OK 

Not  OK 

3a 

2b 

0 

40 

2 

3b 

2b 

0 

40 

2 

3d1 

2c/c 

0 

41 

1 

3d2 

2c /c 

1 

39 

2 

3d  3 

2c/c 

1 

39 

2 

3d4 

2c/c  ' 

1 

39 

2 

3d  5 

2b 

0 

39 

3 

3d6 

2 c/c 

1 

38 

3 

3e 

2b/b 

1 

39 

2 

3f 

2b 

1 

39 

2 

4a 

2b 

0 

41 

1 

4b 

2b 

0 

42 

0 

4c 

2b/b 

0 

41 

1 

4d 

2b 

0 

41 

1 

4e 

2b 

0 

37 

5 

7d 

2b 

0 

42 

0 

9a 

2b 

0 

40 

2 

9b 

2b 

0 

40 

2 

9c 

2b 

0 

40 

2 

9d 

2b 

0 

41 

1 

9g 

1b 

0 

30 

12 

9h 

2b 

0 

42 

0 

13a1 

2b/ la 

0 

41 

1 

13a2 

2b 

0 

39 

3 

13a3 

2b 

3 

35 

4 

13a4 

2b 

4 

33 

5 

13a5 

1b 

4 

34 

4 

13a8 

Ib/b 

4 

34 

4 

13a9 

2b /b 

3 

37 

2 

14a2 

2b 

1 

37 

4 

14b2 

2b 

2 

36 

4 

14c2 

1b 

1 

37 

4 

14d2 

1b 

1 

36 

5 

1 5a  1 

2b 

1 

40 

1 

1 5a2 

2b 

2 

38 

2 

15b2 

1b 

1 

37 

4 

1 5b3 

Ib/b 

1 

39 

2 

15c1 

1b 

1 

38 

3 

1 5c2 

1b 

1 

40 

1 

Supervisor  Ratings  (n=42) 

If  Not  OK.  It  Should  Be 


la  1b  1c  2a  2b  2c 


2 

3 

2 

1 

1 


1 8 1 


1--1- 

2-  — — — 
2 — — _ — 

— — 11  — 
--12- 

— — — 4 — 

2 
3 


continued 


3b  3c 

2 

2 

1 

2 

2 

2 

1 1 

1 2 

1 1 

1 1 


1 

1 

2 

1 

1 1 


1 

2 


[ 

I 

[' 

► 


[ * 

V 

r 


i 

» 
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TABLE  B-28 

Criticality  of  Selected  STS  Job  Elements 

STS  Element 

Number 

Number  of  Supervisor  Ratings  (n=45) 

Overall  Criticality 

Not  Re- 
levant 

Limited 

Relevant 

Impor- 

tant 

Critical 

Not 

Rated 

Mean 

S.D. 

3a 

0 

1 

13 

31 

0 

2.7 

.52 

3b 

0 

1 

16 

28 

0 

2.6 

.53 

3d1 

1 

1 

6 

37 

0 

2.8 

.61 

3d  2 

0 

2 

10 

33 

0 

2.7 

.56 

3d3 

4 

4 

13 

23 

1 

2.3 

.96 

3d4 

1 

2 

15 

26 

1 

2.5 

.70 

3d  5 

0 

1 

9 

35 

0 

2.8 

.48 

3d6 

0 

3 

10 

32 

0 

2.6 

.61 

3e 

0 

3 

16 

25 

1 

2.5 

.63 

3f 

0 

0 

15 

27 

3 

2.6 

.48 

All  3's 

6 

18 

123 

297 

6 

2.6 

4.5 

4a 

0 

7 

30 

8 

0 

2.0 

.58 

4b 

0 

1 

29 

15 

0 

2.3 

.51  „ 

4c 

0 

5 

30 

10 

0 

2.1 

.57 

4d 

0 

2 

30 

13 

0 

2.2 

.53 

4e 

2 

14 

22 

7 

0 

1.8 

.77 

All  4's 

2 

29 

141 

53 

0 

2.1 

.47 

7d 

0 

5 

28 

12 

0 

2.2 

.60 

9a 

0 

1 

26 

18 

0 

2.3 

.53 

9b 

0 

4 

26 

15 

0 

2.2 

.61 

9c 

0 

1 

24 

20 

0 

2.4 

.54 

9d 

0 

3 

26 

16 

0 

2.3 

.59  > 

9g 

0 

2 

26 

17 

0 

2.3 

.56 

9h 

0 

3 

26 

16 

0 

2.3 

.59 

All  9't 

0 

14 

154 

102 

0 

2.3 

.46 

13a1 

3 

8 

24 

9 

1 

1.9 

.81 

13a2 

0 

1 

25 

19 

0 

2.4 

.54 

13a3 

9 

15 

15 

5 

1 

1.4 

.94 

13a4 

18 

17 

6 

2 

2 

.8 

.85 

13a5 

14 

16 

10 

4 

1 

1.1 

.96 

13a8 

5 

18 

16 

6 

0 

1.5 

.87 

13a9 

5 

21 

12 

7 

0 

1.5 

.89 

aii  la's 

54 

96 

108 

52 

5 

1.5 

.67 

continued 
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TABLE  B-28  (continued) 


Number  of  Supervisor  Ratings  (n-45)  Overall  Criticality 

STS  Element 

Nu  mtoer  


Not  Re- 
levant 

Limited 

Relevance 

Impor- 

tant 

Critical 

Not 

Rated 

Mean 

SO. 

14a2 

4 

10 

22 

8 

1 

1.8 

.86 

14b2 

5 

9 

26 

5 

0 

1.7 

.82 

14c2 

20 

15 

8 

1 

1 

.8 

.83 

14d2 

4 

10 

22 

8 

1 

1.9 

.85 

All  14'* 

33 

44 

78 

22 

3 

1.5 

.68 

1 5a1 

3 

6 

24 

12 

_ 0 

2.0 

.83 

1 5a2 

7 

5 

20 

12 

1 

1.8 

1.00 

1 5b2 

1 

4 

29 

11 

0 

2.1 

.65 

1 5b3 

1 

5 

31 

8 

0 

2.0 

.62 

1 5c  1 

1 

5 

28 

11 

0 

2.1 

.67 

1 5c2 

1 

7 

27 

10 

0 

2.0 

.69 

All  IS'* 

14 

32 

1S9 

64 

1 

2.0 

.63 

16a2 

13 

16 

5 

11 

0 

1.3 

1.14 

16b2 

1 

1 

17 

26 

0 

2.5 

.66 

16c2 

0 

11 

27 

17 

0 

2.4 

.53 

16d2 

0 

3 

26 

16 

0 

2.3 

.59 

16e2a 

13 

12 

13 

7 

0 

1.3 

1.06 

All  16’* 

27 

33 

88 

77 

0 

2.0 

.59 

1 7a2 

0 

0 

29 

16 

0 

2.4 

.48 

1 7b2 

0 

0 

18 

27 

0 

2.6 

.50 

1 7c2 

3 

0 

20 

22 

0 

24 

.80 

1 7d2 

6 

9 

12 

18 

0 

1.9 

1.07 

1 7e2a 

1 

1 

26 

17 

0 

2.3 

.63 

17e2b 

5 

1 

20 

19 

0 

2.2 

.94 

17f2a 

0 

0 

17 

28 

0 

2.6 

.49 

17f2b 

3 

4 

14 

24 

0 

2.3 

.90 

1 7f2c 

9 

6 

18 

12 

0 

1.7 

1.07 

17f2d 

4 

3 

18 

20 

0 

2.2 

.92 

1 7f2e 

2 

2 

23 

17 

1 

2.3 

.75 

17g2 

3 

3 

21 

18 

0 

2.2 

.84 

17h2a 

0 

1 

25 

19 

0 

2.4 

.54 

1 7h2b 

2 

2 

24 

17 

0 

2.2 

.74 

17i2 

2 

3 

28 

12 

0 

2.1 

.71 

All  17'* 

40 

35 

313 

286 

1 

2.3 

.48 

TABLE  B-29 

Impact  of  Training  Deficiency  of  Selected  STS  Job  Elements 

Weights  used  for  impact  score  shown  in  parentheses 


Supervisor  Ratings/Type  of  Impact 


STS  Element 
Number 

Task 

Not  Re- 
quired 
(0) 

No  Im- 
pact 
(11 

No  Up 
grading 
(2) 

Delayed 

Up 

grading 

(2) 

Time  of 
Others 
121 

System 

Relia- 

bility 

(2) 

System 

Down 

Time 

(2) 

Produc- 

tion 

Delays 

(2) 

Weighted 

Impact 

Score 

3a 

0 

4 

8 

8 

19 

13 

7 

8 

1.94 

3b 

0 

4 

7 

9 

20 

10 

6 

5 

1.93 

3d? 

1 

5 

7 

9 

17 

9 

7 

5 

1.88 

3d2 

1 

6 

6 

11 

17 

8 

4 

4 

1.86 

3d3 

16 

2 

5 

3 

10 

5 

4 

4 

1.31 

\ 

3d  4 

4 

7 

7 

7 

14 

7 

4 

4 

1.72 

3d  5 

5 

10 

7 

15 

11 

7 

5 

5 

1.89 

3d6 

3 

6 

8 

8 

16 

8 

6 

5 

1.80 

3e 

0 

10 

4 

9 

18 

8 

4 

5 

1.83 

3f 

0 

6 

5 

7 

20 

12 

11 

8 

1.91 

All  3's 

26 

55 

67 

78 

166 

91 

60 

53 

1.82 

4a 

2 

4 

5 

11 

21 

6 

4 

4 

1.86 

4b 

0 

3 

8 

11 

20 

7 

4 

5 

1.86 

4c 

2 

4 

5 

11 

20 

7 

4 

5 

1.86 

4d 

2 

2 

5 

12 

21 

6 

5 

6 

1.90 

4e 

3 

5 

3 

11 

20 

5 

4 

4 

1.80 

Ail  4's 

9 

18 

26 

56 

102 

31 

21 

24 

1.88 

7d 

1 

4 

5 

9 

24 

7 

2 

3 

1.89 

9a 

1 

2 

7 

11 

20 

12 

7 

8 

1.94 

9b 

4 

2 

4 

11 

16 

13 

5 

5 

1.83 

9c 

1 

4 

7 

9 

18 

14 

6 

5 

1.91 

9d 

1 

3 

7 

10 

20 

14 

7 

7 

1.93 

f ' 

9g 

1 

4 

7 

9 

21 

14 

6 

7 

1.91 

9h 

1 

3 

8 

9 

24 

10 

5 

6 

1.92 

All  9'* 

9 

18 

40 

59 

119 

77 

36 

38 

1.91 

13a1 

3 

6 

7 

9 

19 

5 

4 

4 

1.79 

13a2 

1 

4 

7 

12 

17 

11 

7 

7 

1.91 

a 

13a3 

11 

9 

2 

6 

11 

5 

3 

2 

1.37 

13a4 

22 

8 

0 

2 

3 

1 

1 

2 

.67 

13a5 

13 

10 

2 

5 

8 

2 

2 

3 

1.20 

13a8 

14 

9 

2 

3 

8 

3 

2 

2 

1.14 

. • 

13a9 

7 

12 

1 

7 

13 

4 

2 

2 

1.46 

y 

All  13'* 

71 

58 

21 

44 

79 

31 

21 

22 

1.42 

TABLE  29  (continued) 


STS  Element 
Number 

Task 

Not  Re- 
quired 
(0) 

Supervisor  Ratings/Type  of  Impact 

— 

Average 

Weighted 

Impact 

Score 

No  Im- 
pact 
(1) 

No  Up- 
grading 
(21 

Delayed 

Up- 

grading 

(2) 

Time  of 
Others 
(2) 

System 

Relia- 

bility 

(21 

System 

Down 

Time 

(2) 

Produc- 

tion 

Delays 

(21 

1 4a2 

6 

8 

3 

10 

14 

8 

3 

3 

1 .64 

14b2 

5 

9 

2 

9 

15 

7 

4 

3 

1.65 

14c2 

20 

6 

2 

4 

5 

3 

2 

2 

.95 

14d2 

2 

10 

2 

10 

17 

7 

4 

3 

1.75 

All  14'$ 

33 

33 

9 

33 

51 

25 

13 

11 

1.52 

15al 

3 

3 

6 

8 

22 

9 

7 

8 

1.86 

1 5a2 

3 

2 

4 

11 

23 

8 

9 

7 

1 88 

1 5b2 

1 

2 

6 

10 

24 

9 

8 

7 

1.94 

1 5b3 

1 

2 

5 

10 

26 

10 

7 

6 

1.94 

1 5c  1 

1 

2 

6 

9 

24 

9 

6 

5 

1.94 

15c2 

1 

2 

6 

9 

25 

9 

5 

5 

1.94 

All  15’$ 

10 

13 

33 

57 

144 

54 

42 

38 

1.92 

16a2 

15 

7 

5 

6 

5 

2 

4 

3 

1.21 

16b2 

2 

7 

7 

11 

17 

9 

9 

8 

1.84 

16c2 

2 

6 

8 

11 

18 

9 

8 

8 

1.86 

16d2 

2 

6 

6 

12 

18 

8 

8 

7 

1.85 

16e2a 

15 

7 

3 

6 

5 

3 

5 

4 

1.23 

All  16'$ 

36 

33 

29 

46 

63 

31 

34 

30 

1.65 

17a2 

2 

5 

8 

12 

16 

8 

8 

6 

1.86 

1 7b2 

2 

6 

5 

13 

18 

9 

8 

6 

1.85 

17c2 

2 

5 

7 

13 

17 

9 

8 

7 

1.88 

17d2 

11 

6 

4 

8 

9 

4 

3 

2 

1.40 

1 7e2a 

2 

6 

3 

14 

20 

9 

7 

7 

1.85 

1 7e2b 

4 

5 

3 

13 

19 

9 

7 

7 

1.81 

17f2a 

2 

4 

4 

15 

20 

9 

7 

8 

1.88 

17f2b 

6 

4 

3 

12 

17 

8 

5 

7 

1.74 

17f2c 

12 

5 

2 

8 

12 

6 

5 

6 

1.48 

17f2d 

2 

5 

3 

13 

21 

8 

7 

8 

1.87 

1 7f2e 

2 

7 

3 

13 

20 

9 

7 

8 

1.84 

1 7g2 

7 

7 

1 

10 

15 

7 

6 

6 

1.64 

17h2a 

2 

8 

4 

11 

18 

9 

6 

7 

1.82 

1 7h2b 

2 

5 

3 

11 

21 

9 

6 

8 

1.86 

17i2 

5 

4 

2 

12 

19 

8 

5 

6 

1.77 

aii  ir$ 

63 

82 

55 

178 

262 

121 

95 

99 

1.78 
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TABLE  B-30 

Hours  of  Instruction  versus  Task  Criticality 


Task  Criticality 


(0.00-1.8) 


Low  Middle 
(1.90-2.2) 


High  Middle 
12.3-2.4) 


12.5-2.8) 


Hours  of  Instruction 


Less  than  1 
(0) 

More  than  1 , 
Less  than  2 
(1) 

Between  2 and  6 
(2-6) 

13a4,  14c2, 

13a9,  1 6e2a, 

13a8,  14b2, 

4e 

1 7f2c,  1 5a2 

14a2 

1 7g2,  13a1, 

1 7i2,  1 5a 1 

4c,  17d2, 

17f2d,  4d,  1 5c1 , 

17e2b,  1 7h2b, 

1 5c2 

1 5b3,  1 5b2 

3d3,  17f2e, 

17f2b,  9d, 

1 7c2,  1 7h2a, 

16c2 

1 7a2,  9c,  9g 

17e2a,  16d2 

3d1 , 3d2,  3d4, 

17f2a 

1 7b2,  3f 

3d6,  3e 

13a5,  13a3 


4b,  9h,  9a 


16b2,  3a,  3b, 


TABLE  B-31 

Hours  of  Instruction  versus  Impact  of  Training  Deficiency 


Hours  of  Instruction 


Impact  of  Training 
Deficiency 


(0.00-1.69) 


Low  Middle 
(1.70—1 .84) 


High  Middle 
(1.85-1.86) 


(1.87-1.94) 


Less  than  1 
(0) 

More  than  1 , 
Less  than  2 
(11 

Between  2 and  6 
(261 

14c2,  1 7g2 

13a4,  3d3 

16e2a,  17f2c, 
13a9 

13a8,  14b2, 
17d2,  14a2 

17f2e,  4e,  3e, 
13a1,3d4,  3d6 

17i2,  17f2b 

17h2a,  1 7e2b 

16c2, 4d,  3d 2 

17a2,  15a1, 

1 7b2,  16d2, 
17e2a,  17h2b, 

4c 

1 5c1 , 15c2, 

1 7f2d,  3d1 

9g,  9d,  9c, 

1 7f2a,  15a2 

1 5b2,  15b3, 
17c2,  3f 
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3b,  3d 5,  13a2 


TABLE  B-35 

Hours  of  Instruction  versus  Graduate  Reports 
of  Training  Adequacy 


Hours  of  Instruction 


Training  Adequacy 


(1.6— 1.8) 


Low  Middle 


High  Middle 
( 2.0-2. 1) 


12.2-2.3) 


Not  Measured, 


4e,  13a1, 

1 7f2d,  4d 


16c3,  15c1, 
1 5c2 


3d6,  3e,  13a4, 
1 7f2e,  17g2 


More  than  1 , 
Less  than  2 
(1) 

Between  2 and  6 
(2-6) 

16e2a,  17f2c 

17e2a,  17e2b, 
17h2b,  16d2 

17f2b 

14a2,  17b2 

9d,  1 7a2, 
17f2a,9c,  9g, 
15a2 

1 5b3 

15a1 

3f,  1 5b  2 

13a9,  1 7i2 

4c,  1 3a8,  14b2, 
17c2,  17d2, 
17h2a 

4a,  4b,  7d,  9b, 


3a,  3b,  3d5,  9a 
13a2 


Overall  Proficiency 


4e,  1 7e2b, 

17f2c,  1 7a2,  9b, 

13a3,  1 5a1 , 1 5a2, 

17h2b,  17f2d, 

16d2,  4d,  4b,  4a, 

9c,  3f,  9h,  3a,  9a, 

16c2 

9d,  7d,  17f2a, 

3b,  13a2,  3d5 

17f2b 

TABLE  B-37 

Overall  Proficiency  versus  Task  Criticality 


TABLE  B-39 

Overall  Proficiency  versus  Graduate  Reports  of 
Training  Adequacy 


Criticality 
versus  Impact 


